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INTRODUCTION: 


The  need  for  behavioral  health  support  during  warrior  reintegration  is  clear  and  compelling:  changes  in 
the  conditions  of  warfare  and  deployment  in  Iraq  and  Afghanistan,  exacerbating  the  stresses  of  war 
(Belasco,  2007;  Bruner,  2006;  Serafino,  2003),  have  combined  with  large  numbers  of  deployed  troops. 
Unfortunately,  there  have  been  too  few  adequately  prepared  behavioral  health  professionals  to  meet  the 
needs  of  our  nation’s  warriors  and  their  families.  This  research  project,  conducted  under  the  auspices  of 
the  USC  School  of  Social  Work,  Center  for  Innovation  and  Research  on  Veterans  and  Military  Families 
(CIR),  aims  to  develop  and  test  methods  for  rapidly  increasing  the  number  of  behavioral  health 
professionals  who  are  prepared  to  effectively  treat  mental  health  challenges  among  active  duty 
servicemembers,  veterans,  and  their  families.  Project  objectives  include:  1)  Development  and  delivery  of 
four  full  online  graduate  military  social  work  courses,  currently  titled:  Military  Culture,  Clinical  Practice  with 
Servicemembers  and  Veterans,  Families  Impacted  by  Military  Service:  Understanding  and  Intervening, 
and  Health  Challenges  for  Wounded  Warriors  and  their  Caregivers  -  along  with  several  related  training 
modules  for  experienced  mental  health  clinicians  -  to  increase  trainees’  knowledge  of  military  mental 
health  issues  compared  with  standard  training;  and  2)  Development  of  two  virtual  patients  and  a  virtual 
family  member  to  approximate  “real  world”  clinical  interactions  and  facilitate  more  rapid  acquisition  of 
practice  experience.  The  educational/training  intervention  is  being  developed  in  an  innovative  and 
evidence-based  manner,  using  state-of-the-art  knowledge  and  technologies,  including  empirically- 
supported  intervention  content,  and  a  rigorous  process  of  iterative  testing  and  refinement.  Ultimately,  its 
impact  on  trainee  knowledge,  skill,  and  sense  of  competence  -  along  with  ability  to  engage  and  retain 
clients  and  to  develop  a  strong  therapeutic  alliance  -  will  be  examined  in  a  series  of  randomized, 
controlled  trials. 
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BODY: 


As  described  in  the  project  proposal  narrative  and  approved  Statement  of  Work,  this  project  aims  to 
develop  and  test  an  education  and  training  intervention  designed  to  rapidly  increase  the  number  of 
behavioral  health  professionals  prepared  to  effectively  treat  mental  health  challenges  among 
servicemembers,  veterans,  and  their  families.  The  intervention  comprises  two  distinct,  yet  interrelated 
dimensions:  a  structured,  multidimensional  course  curriculum  and  a  content-specific,  culturally-relevant 
Virtual  Patient  Training  Environment  (VPTE),  built  using  artificially-intelligent  virtual  human  technology. 
Due  to  the  complexity  and  extent  of  effort  involved  in  developing  such  an  intervention,  the  first  year 
activities  of  the  project  focused  largely  on  intervention  development,  in  preparation  for  full  implementation 
of  the  intervention  and  evaluation  of  its  impact  later  in  the  project.  Following  (in  grey)  is  the  Phase  II 
portion  of  the  original,  approved  Statement  of  Work,  outlining  the  objectives  to  be  accomplished  during 
the  first  1 8  months  of  project  funding.  The  accomplishments  for  each  portion  of  the  SOW,  and  any 
related  difficulties  that  were  encountered,  will  be  described  in  turn  on  the  pages  that  follow. 


Approved  Statement  of  Work 

Phase  II:  Development  and  iterative  refinement  of  the  training  curriculum  and  VPTE  based  on 
findings  from  initial  trainees  and  VPTE  beta-testers;  Pilot  Randomized  Study  Examining  Impact 
of  Mil-SW  Curriculum  on  Clinical  Outcomes  for  Sendee  Members  with  Depression  and  PTSD 

1 .  Revolutionize  curriculum  content  and  delivery 

1.1.  Using  content  experts,  develop  four  online  courses  (Military  Culture,  Trauma  and  PTSD: 
Evidence  Based  Treatment,  Clinical  Practice  with  Military  Families,  and  Health  Challenges 
for  Returning  Veterans  and  Their  Families)  and  accompanying  brief  curriculum  modules, 
refining  as  needed  based  on  quantitative  and  qualitative  data  from  test  trainees,  including 
knowledge  and  skill  development,  sense  of  competence,  and  curriculum  coherence 

1 .2.  Enhance  online  courses  with  multimedia  and  virtual  reality  online  components 

1.3.  Conduct  a  pilot  randomized  study  to  examine  clinical  outcomes  in  the  treatment  of  service 
members  and  veterans  with  PTSD,  using  experienced  therapists  who  have  completed  Mil- 
SW  training  modules  as  compared  to  those  who  have  not 

2.  Develop  and  implement  VPTE  with  specific  Virtual  Patient  (VP)  and  Virtual  Family  (VF) 

scenarios 

2.1.  Develop  VP,  and  subsequently  a  Virtual  Family  (VF)  (extending  into  Phase  III),  to  address 
learning  needs  related  to  military  culture,  deployment  cycle,  PTSD  and  secondary 
traumatization 

2.2.  Iteratively  test  and  refine  VP  and  VF,  based  on  quantitative  and  qualitative  data  from 
trainees,  including  sense  of  realism  in  communicating  with  VP/VF,  authenticity  of  VP/VF, 
and  sense  of  immersion  in  VP/VF  experience  (ultimately  to  include  diversity  in  culture,  age 
and  gender) 


As  described  in  the  Phase  II  summary  statement,  project  activities  over  the  first  18  months  are  focused  on 
two  primary  objectives:  1)  developing  and  iteratively  refining  the  training  curriculum  intervention  (both  the 
course  curricula  and  the  VPTE)  based  on  findings  from  initial  trainees  and  VPTE  beta-testers;  and  2) 
conducting  a  pilot  randomized  study  to  examine  the  impact  of  the  training  curriculum.  The  former  was  the 
principal  focus  of  Project  Year  1,  and  is  the  area  wherein  most  of  the  project  accomplishments  were 
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achieved  in  the  first  year.  (The  pilot  study  was  not  scheduled  to  occur  in  the  first  12  months  and  has  not 
yet  been  conducted;  it  will  be  discussed  in  a  future  report.) 


Development  of  Four  Online  Graduate  (MSW)  Courses 

Substantial  gains  were  made  in  Year  1 ,  with  regard  to  developing  the  four  online  graduate  (MSW) 
courses.  Extensive  revisions  were  made  to  the  existing  on-ground  course  syllabi,  the  on-ground  courses 
were  evaluated,  and  production  for  online  delivery,  involving  yet  further  course  revision  and  refinement, 
was  initiated.  Subsequently,  two  of  the  courses  -  Military  Culture  and  Clinical  Practice  with 
Servicemembers  and  Veterans  -  were  rolled-out  online,  and  a  large-scale  comparison-group  study  of  the 
MSW  course  curriculum  was  launched.  Further  details  related  to  each  of  the  major  activities  and 
accomplishments  are  provided  in  the  following  paragraphs. 

Extensive  Revision  of  Existing  On-ground  Course  Syllabi 

Early  in  the  project  year,  personnel  completed  extensive  revisions  to  four  existing  on-ground  MSW  course 
syllabi,  using  reviews  of  current  scholarly  literature,  expert  consultants,  findings  from  a  CIR-conducted 
survey  of  348  military  behavioral  health  providers,  and  the  Council  on  Social  Work  Education’s  Advanced 
Social  Work  Practice  in  Military  Social  Work  guidebook.  During  the  revision  process,  it  was  determined 
that  the  Trauma  and  PTSD:  Evidenced  Based  Treatment  course  should  be  somewhat  more  broad  and 
inclusive  of  related  issues  such  as  Depression  and  substance  abuse  than  was  originally  conceptualized; 
hence,  the  course  was  renamed  Clinical  Practice  with  Servicemembers  and  Veterans.  At  this  time,  lead 
course  faculty  met  with  the  School’s  curriculum  consultant  to  identify  learning  outcomes  that  align  with  the 
School’s  curriculum  standards,  and  to  plan  for  the  conversion  of  each  to  online  courses. 

Evaluation  of  Existing  MSW  Courses 

A  questionnaire  regarding  the  MSW  program’s  Military  Social  Work  subconcentration  (Appendix  A), 
including  the  on-ground  courses  as  they  were  delivered  prior  to  any  revisions  by  the  project  team,  was 
administered  to  graduating  students  in  Spring  2011,  shortly  after  they  had  completed  the  course 
sequence.  Twenty-two  graduates  ultimately  completed  the  questionnaire. 

In  general,  the  respondents’  perceptions  of  the  Military  Culture  course  (again,  as  delivered  prior  to  the 
project  team’s  syllabus  revisions)  were  mixed.  Approximately  the  same  percentage  of  respondents 
disagreed  that  the  breadth  and  depth  of  the  course  were  appropriate  (40-50%)  as  did  agree  (40-50%). 
Similarly,  those  disagreeing  that  the  course  positively  impacted  their  practice  or  increased  their 
professional  competence  (35-40%)  were  very  slightly  higher  than  those  agreeing  (30-35%).  The  highest 
percentage  of  agreement  (55%)  was  reported  in  response  to  the  course  being  well  organized  and  flowing 
logically. 

Respondent  perceptions  of  the  two  practice-focused  courses,  Treating  Trauma  and  Post-Traumatic 
Stress  and  Clinical  Practice  with  Military  Families:  Understanding  and  Intervening,  were  decidedly  more 
positive,  particularly  for  the  former.  With  regard  to  Treating  Trauma,  at  least  90%  of  respondents  agreed 
that  the  course  breadth  and  depth  were  appropriate,  and  that  the  course  positively  impacted  their  practice 
and  increased  their  professional  competence.  The  lowest  percentage  of  agreement,  though  still  a 
moderately  strong  81%,  was  in  response  to  the  course  being  well-organized  and  flowing  logically.  In 
response  to  the  questions  regarding  Clinical  Practice  with  Military  Families,  similarly  high  agreement  was 
found  with  regard  to  appropriateness  of  the  course  breadth  and  depth  (91%  and  87%,  respectively). 
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Agreement  that  the  course  had  positively  impacted  the  respondents’  professional  practice  and  increased 
their  professional  competence  was  somewhat  lower  than  for  the  Treating  Trauma  course:  77%  and  72%, 
respectively.  Finally,  similar  to  the  Treating  Trauma  course,  77%  of  respondents  agreed  that  the  course 
was  well-organized  and  flowed  logically. 

The  Health  Challenges  for  Wounded  Warriors  and  their  Caregivers  course  was  not  offered  during 
Academic  Year  2010-201 1.  Thus,  questions  regarding  this  course  were  not  included  in  the  questionnaire. 

Considering  the  nature  of  the  criticisms  offered  by  the  respondents  in  relation  to  the  changes  made  to  the 
course  syllabi  (and  the  rationale  for  those  changes),  it  appears  that  many  of  the  reported  course 
weaknesses  were  likely  addressed  during  the  syllabus  revision  process.  For  example,  substantial 
changes  were  made  to  the  1-  academic  credit  Military  Culture  course,  including  revision  of  topics  and 
instructional  methods,  due  to  the  conclusion  that  the  content  would  benefit  from  some  topic  adjustments 
and  content  reorganization.  This  assertion  will  be  tested  in  a  large-scale  comparison-group  study  of  the 
MSW  course  curriculum,  to  be  discussed  later  in  this  report. 

Course  Production  and  Preparations  for  Online  Delivery 

Early  in  the  calendar  year,  in  cooperation  with  project  personnel  and  lead  course  faculty  for  the  respective 
courses,  2tor,  the  School  of  Social  Work’s  technology  partner  responsible  for  delivery  and  marketing  of 
the  online  MSW  curriculum,  initiated  production  of  the  online  versions  of  the  MSW  courses.  Two  of  the 
online  MSW  Military  Social  Work  courses  were  completed  and  subsequently  launched  online  on  July  18th: 
Military  Culture  as  a  Workplace  Environment,  and  Treating  Trauma  and  Post-Traumatic  Stress  (it  has 
since  received  approval  for  its  new  name:  Clinical  Practice  with  Servicemembers  and  Veterans).  In 
preparation  for  the  launch,  CIR  project  team  members  met  with  seven  faculty  instructors  who  would  be 
teaching  the  Treating  Trauma  course  (both  on-ground  and  online),  to  review  the  syllabus  and  prepare  the 
new  instructors  on  the  course  content  and  materials.  The  Military  Culture  course  is  a  one-unit  self- 
directed  online  course  with  no  live  instruction,  so  CIR  did  not  need  to  facilitate  a  meeting  with  an 
instructor  for  that  course.  The  third  online  MSW  course,  Clinical  Practice  with  the  Military  Family,  is 
scheduled  to  launch  on  December  5,  201 1 ,  with  Health  Challenges  for  Wounded  Warriors  and  their 
Caregivers  scheduled  to  launch  in  August  2012.  These  courses  will  also  be  reviewed  in  detail  by  the  CIR 
project  team. 

Though  the  online  graduate  courses  were  designed  to  mirror  their  on-ground  counterparts,  conversion  to 
an  online  delivery  format  was  a  time-  and  labor-  intensive  process,  involving  writing  of  content  for  the 
online  modules,  finding  video  media  clips  to  be  used  in  the  online  classrooms  (with  the  support  of 
Wounded  Warriors’  film  archives),  and  filming  lecture  segments  to  be  played  online.  While  large-scale 
evaluation  of  the  online  courses  has  not  yet  been  completed  (and  will  be  discussed  later  in  this  report), 
early  anecdotal  evidence  suggests  that  the  content  of  the  online  course  versions  is  presented  in  a  more 
stimulating  and  appealing  way  than  has  traditionally  been  available  in  online  courses,  and  that  the  on¬ 
ground  courses  will  likely  be  strengthened  by  the  work  done  for  -  and  lessons  learned  from  -  their  new, 
online  counterparts. 

Ongoing  Course  Refinement 

Throughout  the  year,  project  personnel  continued  to  work  closely  with  School  of  Social  Work  faculty  on 
revisions  to  the  syllabi  for  the  four  on-ground  Military  Social  Work  MSW  courses.  Specific  attention  was 
given  to  the  Health  Challenges  course,  which  underwent  relatively  greater  revision,  largely  triggered  by 
evaluation  feedback  from  the  Health  Challenges  on-ground  CEU  course.  The  revisions  focused  on 
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eliminating  redundancies  with  the  other  courses,  and  honing  in  on  caregiver  issues/needs,  as  well  as 
issues/needs/services  as  related  to  persons  with  physical  disabilities.  Team  members  worked  closely 
with  the  new  instructor  for  the  Health  Challenges  course,  connecting  her  with  the  curriculum  consultant  to 
promote  collaboration.  The  revised  on-ground  courses,  based  in  part  on  student  data  collected  by  the 
project  team,  will  be  implemented  in  Fall  2011  and  Spring  2012.  In  addition,  based  on  student  response 
to  the  on-ground  course,  the  school  will  decide  whether  to  convert  this  course  to  online. 

After  the  online  courses  were  launched,  CIR  project  team  members  reviewed  the  online  content  for  the 
Military  Culture  course,  and  submitted  feedback  to  the  course  leads  and  technology  leads.  This  feedback 
was  reviewed,  and  certain  changes  were  implemented  immediately  to  the  online  content.  A  review  of  the 
online  Treating  Trauma  content  is  in  process,  and  a  more  thorough  revision  of  the  online  courses  is 
scheduled  to  occur  in  the  coming  months. 

Testing  and  Evaluation  of  MSW  Courses 

In  preparation  for  a  large-scale  evaluation  of  the  MSW  Military  Social  Work  curriculum,  project  personnel 
spent  Quarters  2  and  3  solidifying  measures  for  evaluating  knowledge,  skill,  and  self-efficacy  gains; 
finalizing  recruitment  and  data  collection  protocols;  and  securing  the  necessary  human  subjects 
approvals  through  the  USC  Institutional  Review  Board  and  USAMRMC’s  Human  Research  Protection 
Office  (HRPO). 

Development  of  a  specialized  clinical  skill  assessment  measure.  In  addition  to  development 
of  the  other  measures,  a  particularly  strong  project  accomplishment  is  the  development  of  an  original 
measure  of  clinical  practice  skill  as  related  to  military  populations,  as  demonstrated  clinical  skill  (aka 
Procedural  Competence;  Bogo,  Regehr,  Logie,  Katz,  Mylopoulos,  &  Regehr,  2011)  has  not  commonly 
been  assessed  within  behavioral  health  practice,  and  even  more  rarely  so  in  social  work.  CIR's  clinical 
skill  assessment  tool  was  developed  to  assess  selected  military-relevant  clinical  skills  in  a  time  and  cost- 
efficient  way,  as  the  original  plan  of  assessing  performance  through  interviews  with  the  Virtual  Patient 
(VP)  was  not  possible  due  to  the  extended  timeline  for  VP  development  (to  be  discussed  later  in  this 
report).  Traditionally,  standardized  patients  have  been  used  to  measure  clinical  skill.  As  part  of  the 
Military  Social  Work  curriculum  evaluation,  the  project  team  needed  to  assess  military-relevant  clinical 
skill  in  over  100  students  at  three  time  points  during  the  academic  year.  Relying  on  standardized  patients 
would  have  been  both  time  and  cost  prohibitive.  The  CIR  measure  is  internationally  informed;  the  team 
relied  heavily  on  the  literature  of  the  leaders  in  social  work  practice  assessment  at  the  University  of 
Toronto  as  well  as  that  regarding  a  standardized  video  measure,  developed  by  a  team  of  researchers  in 
the  Netherlands,  for  use  in  measuring  skill  development  among  medical  students.  The  CIR  measure  is 
computer  administered  and  asks  trainees  to  respond  to  14  video  clips  of  a  veteran/therapist  interaction. 
(Please  see  http://www.voutube.com/playlist?list=PL07CAF37BCDE5A0D0  for  the  video  clips  depicting 
the  clinical  interactions.)  Responses  are  scored  in  the  domains  of  rapport  building,  appropriate 
communication  and  cultural  awareness,  unique  issues  relevant  to  military  context,  legal  and  ethical  issues 
relevant  to  military  clients,  and  culturally-informed  client  assessment.  A  validation  study  of  the  measure  is 
underway,  and  responses  provided  by  students  as  part  of  the  assessment  will  be  used  towards  further 
development  of  the  VP. 

Large-scale  comparison-group  test  of  the  military  social  work  curriculum.  Documentation 
for  the  comparison-group  test  of  the  Military  Social  Work  MSW  curriculum  was  submitted  to  the  USC 
Institutional  Review  Board  (IRB)  in  July  and  the  project  was  granted  exempt  status  shortly  thereafter.  In 
August,  USAMRMC’s  ORP  also  issued  exempt  status  approval.  Early  in  Quarter  4,  the  CIR  team 
finalized  the  script  for  its  original,  video-enhanced  Clinical  Skill  Assessment  (CSA)  measure  of  military- 
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related  practice  skills.  Behavioral  health  clinicians  experienced  with  military  populations  and  veterans 
provided  feedback  to  revise  both  the  script  and  the  accompanying  prompts  built  into  the  measure.  In 
August,  a  professional  film  crew  shot  and  edited  footage  of  the  veteran  client  and  therapist  interaction 
featured  in  the  CSA.  An  OIF  veteran-actor  donated  his  time  to  the  filming  necessary  to  complete  the  CSA. 
In  addition,  all  of  the  baseline  measures  for  the  study  were  incorporated  into  the  online  survey  delivery 
system,  Formsite,  selected  for  its  capacity  to  display  the  video  clips  embedded  in  the  CSA. 

MSW  students  were  recruited  for  the  comparison-group  study  in  late  August  and  early  September. 
Students  enrolled  in  the  military  subconcentration  of  the  MSW  program  were  recruited  at  a  day-long 
orientation  event  held  the  week  before  courses  began.  Students  consenting  to  participate  completed  the 
measures  in  a  classroom  near  the  project  offices.  Sixteen  computer  stations  were  set  up  with 
headphones  that  students  used  to  complete  the  online  baseline  measures,  which  included  the  CSA. 
Students  were  also  given  the  option  to  bring  their  own  laptop  to  complete  the  measures.  The  33  students 
who  completed  the  baseline  measures  at  the  orientation  event  received  their  study  incentives  (i.e.,  $30 
gift  card  to  Starbucks,  iTunes,  Target  or  Amazon)  immediately  after  completing  the  measures.  A  total  of 
43  students  were  ultimately  enrolled  into  the  study  as  part  of  the  on-ground  Military  Social  Work  group. 

Students  were  recruited  for  the  on-ground  comparison  group  (i.e.,  MSW  students  who  are  not  enrolled  in 
the  military  subconcentration)  on  campus  during  the  first  week  of  classes,  during  a  two-hour  “universal 
break”  within  the  School  of  Social  Work.  Consenting  students  were  assigned  participant  ID  numbers  and 
given  the  web  address  where  to  complete  the  evaluation  measures.  Students  completed  the  baseline 
evaluation  measures  at  their  convenience  during  the  first  two  weeks  of  classes  and  received  their  study 
incentives  in  the  mail.  A  total  of  31  students  enrolled  in  the  study  as  part  of  the  non-  Military  Social  Work 
comparison  group. 

Finally,  students  were  recruited  for  the  online  comparison  group  -  MSW  students  enrolled  in  the  in  the 
military  subconcentration  through  the  Virtual  Academic  Center,  the  School  of  Social  Work’s  virtual 
campus  for  students  taking  their  courses  through  a  web-based  interface  -  via  email.  Consenting  students 
were  assigned  participant  ID  numbers  and  given  the  web  address  where  to  complete  the  evaluation 
measures.  As  with  the  on-ground  comparison  group,  students  completed  the  baseline  evaluation 
measures  at  their  convenience  during  the  first  two  weeks  of  classes  and  received  their  study  incentives  in 
the  mail.  The  online  Military  Social  Work  student  sample  consists  of  seven  student  enrollees. 

In  total,  81  students  have  completed  baseline  measures  for  the  MSW  curriculum  evaluation.  The  dataset 
is  currently  being  cleaned.  Recruitment  of  PhD  student  scorers  for  coding  the  open-ended  responses  and 
CSA  responses  is  underway.  As  the  dataset  is  being  cleaned,  a  validation  study  of  the  CSA  is  in 
preparation.  Recruitment  of  advanced  behavioral  healthcare  providers  to  participate  in  this  study  is 
underway. 


Development  and  Refinement  of  Brief  Continuing  Education  (CE)  Curriculum  Modules 

Shortly  after  completing  the  extensive  revisions  to  the  four  MSW  courses,  development  of  the 
accompanying  brief  Continuing  Education  (CE)  curriculum  modules  for  advanced  behavioral  health 
providers  was  initiated.  The  decision  was  made  to  first  develop  the  CE  curriculum  as  a  set  of  four  on¬ 
ground  courses,  replicating  the  titles  and  overall  topics  of  the  MSW  courses,  with  their  online  counterparts 
slated  for  development  after  a  test  administration  and  subsequent  refinement  of  the  on-ground  courses. 
The  four-course  series  of  on-ground  courses  was  rolled-out  at  the  end  of  Guarter  2,  with  the  four  courses 
being  offered  and  evaluated  in  sequence  over  a  3-month  period.  After  completing  a  revision  of  the 
courses  based  on  the  evaluation  findings,  Year  1  closed  with  preparations  being  made  for  a  second 
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offering  of  the  four  on-ground  courses,  scheduled  to  begin  on  October  1.  The  major  project  activities  and 
accomplishments  related  to  the  SOW  objectives  for  brief  curriculum  modules  will  be  detailed  in  the 
following  paragraphs. 

CE  Course  Development  and  Test  Delivery 

Development  of  the  brief  CE  courses  was  based  largely  on  the  results  of  a  national  survey  of  military 
behavioral  health  providers  conducted  by  CIR  in  Summer  2010.  The  courses,  designed  for  post-graduate 
level  behavioral  health  providers,  were  designed  to  provide  more  advanced  content  than  do  the  MSW 
courses.  Rooted  in  the  most  current  scholarly  literature,  the  CE  courses  include  empirically-supported 
interventions  and  evidence-based  practice  approaches,  focused  on  select  topics  of  most  perceived  value 
to  advanced  behavioral  health  providers.  Approval  was  secured  through  the  California  state  licensing 
board  to  offer  Continuing  Education  Units  (CEUs)  for  Licensed  Clinical  Social  Workers  (LCSWs)  and 
Marriage  and  Family  Therapists  (MFTs)  who  complete  the  courses;  this  designation  often  allows  LCSWs 
and  MFTs  from  other  states  to  receive  credit  for  the  courses,  as  well.  Additional  continuing  education 
certifications  (e.g.,  for  Continuing  Medical  Education  credits  [CMEs]  for  RNs  or  MDs,  or  CEUs  for 
Psychologists  or  Licensed  Professional  Counselors)  will  be  sought  as  needed,  based  on  trainee  demand. 

In  Guarter  2,  the  first  two  completed  CE  courses  were  offered  in  a  classroom  setting  for  testing  and 
evaluation  purposes.  Syllabi,  presentation  slides,  and  participant  resource  binders  were  developed  and 
delivered  for  the  Military  Culture  and  the  Clinical  Practice  with  Servicemembers  and  Veterans  courses. 
Expert  veteran,  civilian,  and  military  reserve  component  instructors  from  the  USC  School  of  Social  Work 
faculty,  the  Department  of  Veterans  Affairs,  and  the  Center  for  Deployment  Psychology  were  contracted 
to  teach  the  courses.  The  third  and  fourth  brief  CE  courses  were  completed  and  delivered  in  Ouarter  3, 
again  primarily  for  purposes  of  testing  and  evaluation.  As  with  the  first  two  courses,  syllabi,  presentation 
slides,  and  participant  resource  binders  were  developed  and  delivered  for  the  Families  Impacted  by 
Military  Service:  Understanding  and  Intervening  and  Health  Challenges  for  Wounded  Warriors  and  their 
Caregivers  courses.  Expert  veteran  and  civilian  instructors  were  contracted  to  teach  the  courses  from  the 
USC  School  of  Social  Work  faculty,  the  Department  of  Veterans  Affairs,  and  the  Indiana 
University/Purdue  University  Indianapolis  School  of  Social  Work  faculty.  The  highlight  of  the  Health 
Challenges  course  was  a  special  speaker  -  the  first  known  TBI  patient  from  OIF. 

Testing  and  Evaluation  of  CE  Curriculum  Modules 

Based  on  the  project  team’s  review  of  relevant  literature,  an  evaluation  measure  informed  by  the 
Kirkpatrick  Model  of  Training  Evaluation  (Kirkpatrick,  1996)  and  the  input  of  expert  education  evaluation 
consultants  was  developed  for  purposes  of  evaluating  the  initial  delivery  of  the  CE  courses.  A 
multidimensional  post-test  questionnaire  (Appendix  B)  examined  respondents’  affective  reaction 
(participants’  satisfaction  and  enjoyment  of  the  course)  and  utility  judgments  (participants’  beliefs  about 
how  much  they  learned  and  how  they  plan  to  use  what  they  learn  in  their  practices  with  veterans  and 
military  families).  For  this  purpose,  24  open-  and  closed-ended  questions,  consistent  across  courses 
except  for  a  single  question,  were  accompanied  by  items  seeking  to  obtain  a  brief  demographic  profile  of 
respondents. 

A  20-item  pre/post  measure  of  content-specific  knowledge  was  also  developed  for  each  course  to 
measure  learning  (knowledge  acquired,  skills  improved,  or  attitudes  changed  due  to  training)  (Appendices 
C  through  F).  USC’s  IRB  was  consulted  regarding  the  planned  course  evaluation,  and  they  confirmed 
that  course  evaluation  practices  are  considered  Not  Human  Subjects  Research.  Thus,  no  additional 
Human  Subjects  approvals  were  sought  for  this  particular  activity. 
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Each  of  the  four  CE  courses  was  subsequently  evaluated  during  its  classroom-based  delivery  during 
either  the  second  or  third  quarter,  with  prospective  attendees  being  offered  free  enrollment  in  exchange 
for  participating  in  the  initial,  test  run  of  the  course  (i.e.,  understanding  that  they  would  be  expected  to 
provide  feedback  and  to  complete  a  set  of  pre-  and  post-test  measures).  Appendices  G  through  J  offer 
detailed  reports  of  the  respective  respondent  characteristics,  evaluation  results,  and  accompanying 
recommendations  for  each  of  the  courses.  In  general,  findings  from  all  four  course  evaluations  indicate 
that  providers  learned  from  the  courses,  were  highly  satisfied  with  them,  and  planned  to  implement  the 
information  they  learned  in  their  practice.  Brief  highlights  will  be  presented  below,  in  turn. 

Military  Culture.  Fifteen,  mostly  White  (73%)  and  female  (91%),  participants  from  the  initial,  10- 
hour  Military  Culture  course  completed  the  course  evaluation  questionnaire.  Results  indicated  that 
participants  were  highly  satisfied  with  the  Military  Culture  course.  The  average  response  to  items  on  the 
affective  reaction  scale  was  4.25  out  of  a  possible  5  (range:  3.62  to  4.77),  indicating  a  high  level  of 
satisfaction  with  the  course  content,  delivery,  and  overall  value.  Participants  also  expressed  the  belief 
that  knowledge  gained  in  the  course  will  affect  how  they  approach  their  clinical  practice.  The  average 
response  to  items  on  a  utility  judgments  scale  was  4.26  out  of  a  possible  5  (range:  3.79  to  4.43), 
indicating  a  strong  sense  of  intention  to  apply  course  concepts  to  practice  with  military  clients.  Finally, 
there  was  evidence  of  knowledge  gains  related  to  military  culture,  as  well.  Participants  scored 
significantly  higher  on  the  knowledge  measure  at  the  conclusion  of  the  course  (M  =  13.00,  SD  =  2.16) 
than  at  the  beginning  of  the  course  (M  =  10.23,  SD  =  2.49),  t(12)  =  4.62,  p  <  .05.  No  statistically 
significant  differences  emerged  on  these  three  domains  by  profession  (i.e.,  MFT  or  LCSW),  years  of 
practice  experience,  or  percentage  of  practice  dedicated  to  military  clients.  With  regard  to  course 
refinement,  participants  requested  that  material  be  presented  at  a  more  basic  level  and  that  more 
interactive  training  modalities  be  used  in  instruction.  In  addition,  numerous  participants  requested 
additional  information  on  ethical  considerations  in  practice  when  treating  veterans  and  military  families. 

Clinical  Practice  with  Servicemembers  and  Veterans.  With  regard  to  the  course  on  Clinical 
Practice  with  Servicemembers  and  Veterans,  18  behavioral  health  provider  trainees,  mixed  with  regard  to 
age,  sex,  and  professional  discipline,  completed  the  15  hour  course  and  subsequently  completed  the 
course  evaluation  questionnaire.  In  terms  of  satisfaction  and  enjoyment  of  the  course  (affective  reaction), 
the  average  response  to  the  evaluation  items  was  4.52  out  of  a  possible  5,  (range:  3.91  to  4.91 ), 
indicating  a  high  level  of  satisfaction  with  the  course  content,  delivery,  and  overall  value.  The  most 
favorable  responses  regarded  the  overall  value  of  the  course  and  the  ability  of  the  instructors  to  engage 
with  class  participants.  For  utility  judgments,  the  average  response  to  the  scale  items  was  4.39  out  of  a 
possible  5,  (range:  4.06  to  4.69),  indicating  a  strong  sense  of  intention  to  apply  course  concepts  to 
practice  with  military  clients.  In  terms  of  learning,  participants  were  found  to  have  scored  significantly 
higher  on  the  knowledge  measure  at  the  conclusion  of  the  course  (M  =  14.93,  SD  =  2.09)  than  at  the 
beginning  of  the  course  (M  =  12.93,  SD  =  2.40),  t(15)  =  3.87,  p  <  .05.  This  suggests  that  trainees’ 
knowledge  base  relevant  to  clinical  practice  with  military  populations  increased  in  relation  to  the  15  hours 
of  course  content.  No  statistically  significant  differences  emerged  on  the  three  domains  by  profession, 
experience,  or  amount  of  practice  dedicated  to  military  clients.  With  regard  to  refining  the  course,  survey 
results  indicated  that  instruction  methods  need  to  be  more  balanced  (e.g.,  with  use  of  small  group  or 
discussion  formats).  In  addition,  participants  did  not  report  high  confidence  in  knowing  where  to  turn  for 
further  resources  on  practice  with  veteran  and  military  family  clients. 

Families  Impacted  by  Military  Service:  Understanding  and  Intervening.  Twenty-two  course 
participants,  diverse  with  regard  to  age,  sex,  professional  discipline,  and  length  of  practice,  completed  the 
course  evaluation  questionnaire  for  Families  Impacted  by  Military  Service:  Understanding  and 
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Intervening.  Responses  generally  indicated  that  participants  were  highly  satisfied  with  the  course, 
believed  knowledge  gained  in  the  course  will  affect  how  they  approach  clinical  practice  with  veterans  and 
military  families,  and  gained  content-specific  knowledge  over  the  two  days  of  instruction.  Participant 
ratings  on  the  affective  reaction  scale  items  averaged  4.40  out  of  a  possible  5  (range:  4.19  to  4.76), 
indicating  a  high  level  of  satisfaction  with  the  course  content,  delivery,  and  overall  value.  Overall,  the 
average  response  to  the  utility  judgments  scale  was  4.46  out  of  a  possible  5  (range:  4.32  to  4.68), 
indicating  a  strong  sense  of  intention  to  apply  course  concepts  to  practice  with  military  clients.  Finally, 
with  regard  to  learning,  participants  collectively  scored  significantly  higher  on  the  knowledge  measure  at 
the  conclusion  of  the  course  (65%  correct;  M  =  13.00,  SD  =  2.40)  than  at  the  beginning  of  the  course 
(58%  correct;  M  =  1 1 .58,  SD  =  2.43),  t(1 9)  =  2.48,  p  <  .05.  No  statistically  significant  differences  emerged 
on  these  by  profession,  experience,  or  amount  of  practice  dedicated  to  military  clients.  With  regard  to 
refining  the  course,  participants  requested  that  knowledge  be  presented  at  a  more  basic  level  and  that 
more  interactive  training  modalities  be  used  in  instruction. 

Health  Challenges  for  Wounded  Warriors  and  their  Caregivers.  As  with  the  other  courses, 
evaluation  results  from  the  Health  Challenges  for  Wounded  Warriors  and  their  Caregivers  course  were 
also  promising  and  informative.  The  available  data  on  Levels  1  and  2  of  the  Kirkpatrick  Training 
Evaluation  Model  suggested  that,  for  this  course,  too,  participants  were  highly  satisfied  with  the  course, 
believed  knowledge  gained  in  the  course  would  affect  how  they  approached  their  practice  with  veterans 
and  military  families,  and  demonstrated  gains  in  knowledge  over  the  two  days  of  instruction.  Twenty-five 
course  participants  -  the  largest  and  most  diverse  group  thus  far,  with  regard  to  their  demographic  profile 
-  completed  the  course  evaluation  questionnaire.  In  terms  of  affective  reaction,  the  average  response  to 
the  scale  items  was  4.56  out  of  a  possible  5  (range:  4.19  to  4.91),  indicating  a  high  level  of  satisfaction 
with  the  course  content,  delivery,  and  overall  value.  Items  with  the  most  favorable  responses  inquired 
about  the  overall  value  of  the  course  and  the  importance  of  training  like  this  for  practitioners  working  with 
veterans  or  military  families.  Similarly,  the  average  response  to  the  utility  judgments  scale  was  4.57  out 
of  a  possible  5  (range:  4.26  to  4.74),  indicating  a  strong  sense  of  intention  to  apply  course  concepts  to 
practice  with  military  clients.  The  item  with  the  strongest  endorsement  inquired  about  participants’ 
confidence  finding  additional  resources  and  information  on  working  with  military-related  clients.  Finally, 
results  indicated  that  overall,  participants  scored  significantly  higher  on  the  knowledge  measure  at  the 
conclusion  of  the  course  (78%  correct;  M  =  15.61,  SD  =  1.94)  than  at  the  beginning  of  the  course  (69% 
correct;  M  =  13.72,  SD  =  2.02),  t(17)  =  4.59,  p  <  .0001.  No  statistically  significant  differences  emerged  on 
these  by  profession,  experience,  or  amount  of  practice  dedicated  to  military  clients. 

Planning  for  Continuing  CE  Course  Delivery:  Further  Refinement  and  Marketing 

During  the  fourth  quarter,  a  course  pricing  structure  was  developed,  as  well  as  an  initial  marketing  plan 
for  the  on-ground  CE  courses.  Project  personnel  reviewed  provider  feedback  from  the  previous  iteration 
of  courses.  Incorporating  changes  suggested  by  the  evaluation  findings  and  including  other  updated 
information,  the  project  team  revamped  the  Military  Culture  CE  course  curriculum,  structure,  resource 
binder,  and  slides.  The  project  team  researched  new  topics  to  provide  the  most  relevant  and  current 
information  in  the  slides  and  course  binders.  The  team  also  held  interviews  to  locate  instructors.  In  order 
to  minimize  the  cost  of  course  delivery,  the  project  team  limited  the  search  to  Southern  California. 
Potential  instructors  from  the  USC  School  of  Social  Work,  the  Department  of  Veterans  Affairs,  and 
community-based  organizations  that  serve  veterans  were  interviewed.  Expert  veteran  and  civilian 
instructors  were  selected  and  contracted,  after  which  time  the  project  team  met  with  them  on  a  weekly 
basis  to  plan  for  the  next  course  offering.  A  female  veteran  was  also  selected  to  speak  in  the  course 
about  her  experience  with  deployment  and  being  a  woman  in  the  military.  Project  personnel  marketed 
the  course  to  local  providers  using  web-blasts,  newspaper  ads,  online  calendars,  and  networking 
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capacity.  Targeted  marketing  was  directed  at  community  providers  serving  veterans  in  various  settings. 
The  decision  was  made  to  charge  a  fee  in  order  to  partially  defray  the  costs  of  delivery,  and  scholarships 
were  offered  to  a  select  number  of  community  providers  to  attend  the  course  at  no-charge.  As  Year  1 
came  to  an  end,  planning  for  the  next  7-hour,  on-ground  Military  Culture  CE  course  was  finalized, 
scheduled  to  take  place  in  October  201 1 . 

Preparations  for  Online  Course  Launch 

With  each  of  the  on-ground  CE  courses  delivered  successfully,  project  personnel  met  during  Quarter  3 
with  three  potential  technology  partners  -  2tor,  Embanet,  and  ISV/T3  -  poised  to  create  the  online  CEU 
platform.  Both  2tor  and  Embanet  are  recognized  technology  companies  who  have  existing  relationships 
with  USC,  while  ISV/T3  approached  CIR  in  pursuit  of  a  partnership.  The  project  team  discussed  the 
project’s  technology  needs,  specifications,  and  timeline,  and  subsequently  gathered  proposals  and 
market  analyses  from  all  three  companies.  Ultimately,  the  decision  was  made  not  to  pursue  a  partnership 
with  any  of  the  three  candidate  organizations.  Instead,  the  project  team  chose  to  partner  with  the  USC 
Department  of  Continuing  Education,  which  had  an  existing  collaboration  with  the  New  York  Times 
Knowledge  Network  (NYTKN).  The  partnership  agreement  was  finalized  midway  through  Quarter  4. 

Concurrent  with  the  process  of  selecting  a  technology  partner,  the  project  team  used  evaluation  findings 
from  the  on-ground  CEU  courses  to  inform  revisions  and  improvements  to  both  the  content  and  structure 
of  the  planned  online  courses. 

As  Year  1  came  to  a  close,  the  online  CE  courses  were  being  built  with  NYTKN  technology  staff,  and  will 
be  hosted  on  the  USC/NYTKN  platform.  The  project  team  is  excited  about  the  potential  for  growth  that  is 
provided  by  this  collaboration.  Given  the  wide  marketing  scope  of  both  the  university  and  New  York 
Times,  we  feel  that  the  CIR  courses  will  reach  a  much  wider  and  more  diverse  audience  than  was 
originally  anticipated. 

Thinking  toward  the  future  -  and  the  sustainability  of  CIR’s  CE  course  offerings  -  the  team  is  considering 
feedback  from  attendees  of  our  on-ground  CEU  courses,  as  well  as  from  past  surveys  of  community 
providers,  and  is  focusing  on  building  a  series  of  courses  that  meets  the  needs  of  working  professionals 
in  regards  to  the  amount  of  time  available  for  continuing  education,  the  preference  for  self-directed 
learning,  the  geographic  location  of  the  potential  participants,  price  concerns,  etc.  Project  personnel  are 
also  exploring  ways  through  which  to  create  a  stimulating,  interactive  learning  environment  that  improves 
upon  currently  available  online  education  models.  As  a  result,  the  slate  of  online  course  offerings  will 
diverge  slightly  from  the  on-ground,  4-course  sequence  modeled  after  the  MSW  curriculum.  The  first 
course,  Military  Culture,  will  be  launched  on  December  5,  2011.  Following  this  course,  a  clinical  skill¬ 
building  series,  comprised  of  live  online  seminars  with  content  experts,  focused  on  specific  evidence- 
based  treatment  modalities,  will  be  launched.  In  addition  to  the  courses,  participants  will  have  access  to 
CIR-led  forums,  professional  peer  groups,  webcast  events,  and  other  features  through  the  online  NYTKN 
platform.  We  believe  this  reformatting  of  the  accompanying  brief  curriculum  modules  remains  consistent 
with  the  approved  SOW,  but  is  more  responsive  to  behavioral  health  providers’  needs  and  stands  to 
make  a  greater  impact  on  provider  readiness  (in  terms  of  the  quantity  of  trained  providers)  than  would 
have  been  possible  with  longer  and  less-focused  courses. 
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Ongoing  Course  Evaluation  Activities 


After  completing  the  initial  pre-post  evaluation  of  the  four  on-ground  CE  courses,  project  personnel 
continued  to  review  scholarly  literature  on  best  practices  in  evaluating  professional  education  and  training 
outcomes  and  follow-up  measures  were  developed  to  assess  how  providers  were  using  the  training  in 
their  practice.  A  questionnaire  was  subsequently  sent  to  each  CE  course  participant,  three  months  after 
their  course  participation.  Unfortunately,  the  response  rate  was  not  sufficient  to  render  the  results 
representative  of  the  course  participant  population;  however,  the  information  provided  was  meaningful  for 
purposes  of  further  refining  the  course  content  and  delivery. 

Also,  in  preparation  for  the  second  offering  of  the  on-ground  CE  courses,  evaluation  measures  used  in 
the  Spring  2011  CE  course  evaluations  were  refined.  The  revised  measures  will  also  be  incorporated  into 
the  online  CE  course  delivery  platforms,  in  order  to  conduct  continuous  evaluation  of  those  courses. 


Enhancement  of  Online  Courses  with  Multimedia  Components 

An  integral  objective  outlined  in  the  approved  SOW  is  that  of  enhancing  online  courses  with  multimedia 
and  virtual  reality  online  components.  While  the  virtual  reality-related  functions  will  not  be  embedded  for 
online  training  use  for  some  time  (to  be  discussed  later  in  this  report),  numerous  multimedia  components 
and  functions  have  been  -  and  continue  to  be  -  built  into  the  online  courses  at  both  the  MSW  and  CE 
course  levels.  The  online  MSW  courses  are  delivered  in  a  synchronous  live  classroom,  complete  with 
breakout  group  capability.  Thus,  students  see  and  interact  with  one  another  -  and  with  the  course 
instructor  -  continuously,  much  in  the  way  that  students  in  more  traditional,  on-ground  settings  would. 
Multimedia  features  included  in  the  online  MSW  courses  include  video  lectures  by  instructors 
(accompanied  by  PowerPoint  presentations),  video  clips  of  case  examples,  discussion  boards,  and  links 
to  outside  resources.  For  the  online  CE  courses  under  development,  all  of  the  multimedia  features  just 
mentioned  (i.e. ,  video  lectures  and  case  example  clips,  discussion  boards,  and  links)  are  being 
incorporated,  along  with  incorporation  of  New  York  Times  news  archive  into  the  courses,  and  guest 
contributors  to  discussion  boards  and  groups  (e.g.,  a  New  York  Times  journalist  who  reported  in  the  A 
Year  at  War  series). 


Development  and  Implementation  of  the  Virtual  Patient  Training  Environment  (VPTE) 
with  Specific  Virtual  Patient  (VP)  and  Virtual  Family  (VF)  Scenarios 

The  core  SOW  objectives  related  to  the  Virtual  Patient  Training  Environment  (VPTE)  during  the  first  18 
months  of  the  project  (Phase  II)  pertain  to:  1)  Virtual  Patient  (VP)  development  in  order  to  address 
learning  needs  related  to  military  culture,  deployment  cycle,  PTSD  and  secondary  traumatization;  and  2) 
iterative  testing  and  refinement  of  the  virtual  patient. 

Virtual  Patient  Development 

In  Quarter  1,  in  consultation  with  the  Institute  for  Creative  Technologies  (ICT)  VP  development 
subcontractor  team,  a  basic  character  profile  for  the  first  VP  was  developed.  Planning  was  finalized  for 
an  initial  videotaped  role-playing  session  -  to  include  therapist  and  client  characters  -  allowing  for 
beginning  dialogue  creation  for  the  VP  corpus.  Developers  began  designing  various  aspects  of  the 
VPTE,  including  new  controllers  designed  to  allow  VPs  to  display  emotion  (i.e.,  cry,  blush,  and  change 
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breathing  patterns)  and  have  memory  (i.e.,  to  remember  the  contents  of  previous  clinical  interview 
discussions  and  respond  accordingly). 

In  an  effort  to  develop  a  VP  training  scenario  that  offers  maximum  real-world  applicability  for  students  and 
trainees,  it  was  determined  during  the  second  quarter  that  the  first  VP  would  represent  a  young  Latino 
enlisted  member  of  the  Marine  Corps.  Hence,  a  detailed  case  vignette  representing  the  life  of  “Alamar 
Castilla,”  a  fictitious  Marine,  was  developed  for  use  in  building  the  first  VP  scenario.  To  begin  building  the 
content  for  the  VP,  a  series  of  case  clinical  interview  role-plays  was  launched.  ICT  worked  with  CIR 
project  personnel  to  film  approximately  six  hour-long  role-playing  sessions  involving  clinical  interviews 
between  therapists  (played  by  social  work  graduate  students)  and  Mr.  Castilla  (played  by  OEF/OIF 
veteran  students).  Footage  from  these  sessions  was  transcribed  and  used  by  ICT  to  capture  dialogue  to 
further  develop  the  VP’s  corpus  of  language.  ICT  personnel  began  transitioning  the  various  game  assets 
to  a  new  (Unity  3D)  game  engine,  designed  for  smoother  systems  operation,  and  a  new  Wizard  of  Oz 
(i.e.,  Woz2)  character  control  system  was  developed  to  allow  users  to  control  various  aspects  of  the  VP 
during  training  sessions.  Simultaneously,  student  learning  objectives  for  use  of  the  VP  were  developed  in 
order  to  guide  the  specific  content  required  to  be  embedded  within  the  VP  corpus. 

During  the  third  quarter,  student  and  trainee  skill  development  objectives  were  further  refined  to  guide  the 
specific  dialogue  and  content  needed  for  effective  training  using  the  VP.  The  objectives  were  been  linked 
to  the  course  content,  and  aligned  with  course  faculty’s  learning  objectives  as  well  as  with  the  CSWE 
guidelines  for  Military  Social  Work  Practice.  As  planned,  MSW  student  volunteers  assisted  with  early  VP 
testing  in  April,  and  attention  was  subsequently  focused  largely  on  expansion  of  the  dialogue  corpus  and 
specification  of  the  emotions  to  be  associated  with  particular  dialogue.  Because  of  the  complexity  of  the 
skill  development  objectives,  CIR  project  personnel  anticipated  that  the  VP  will  ultimately  need  breadth 
and  depth  of  relevant  dialogue  sufficient  for  repeated,  hour-long  clinical  interviews  with  trainees.  Thus, 
CIR  project  personnel  became  extensively  involved  in  the  generation  and  refinement  of  VP  dialogue. 

Collaborators  at  technology  partner  ICT  made  several  strides  with  regard  to  the  technical,  backend 
aspects  of  the  VP  technology  this  past  quarter.  They  continued  to  transition  all  game  assets  to  the  Unity 
3D  game  engine,  and  among  other  ongoing  development  activities,  developed  Screen  Mockups  for  user 
interaction  to  the  Training  environment,  continued  to  refine  the  character  graphics,  and  performed  initial 
motion  capture  with  the  art  group  at  ICT  to  capture  a  set  of  animations  that  will  be  used  by  the  characters. 
They  also  worked  with  CIR  project  personnel  to  determine  the  list  of  character  animation  requirements, 
oversaw  the  initial  user  testing  by  MSW  students,  and  developed  a  dialog  testing  tool  to  assist  CIR  project 
personnel  with  developing  the  language  corpus. 

Late  in  the  third  quarter,  a  technology  project  manager,  responsible  for  CIR  oversight  of  VP  development 
and  related  functions,  joined  the  project  team.  The  new  project  manager  brought  to  the  team  a  software 
development  background,  which  none  of  the  existing  CIR  project  team  members  had,  and  with  better 
understanding  of  the  necessary  technology-development  tasks  and  timelines,  urged  the  team  to 
immediately  begin  informal  testing  of  the  VP  dialogue  in  preparation  for  September  classroom 
implementation.  After  a  series  of  test  interviews  conducted  by  CIR  project  team  members  and  MSW 
student  volunteers,  it  was  determined  that  the  VP  lacked  sufficient  dialogue,  and  that  the  current 
Question-Answer  dialogue  engine  would  not  likely  be  able  to  deliver  the  dialogue  complexity  necessary 
for  effectively  training  military  behavioral  health  providers.  As  a  result,  efforts  in  the  fourth  quarter  were 
focused  on  the  exploration  of  alternative  dialogue-generation  systems,  and  on  understanding  the 
feasibility  and  timeline  for  development  of  the  VP  as  it  is  required  to  function  for  training  purposes. 
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After  a  series  of  meetings  with  ICT  personnel,  the  project  team  determined  that  the  VP  dialogue 
complexity  needs  could  be  met  with  the  assistance  of  some  additional  personnel  from  the  Natural 
Language  Dialogue  Group  at  ICT  (see  http://proiects.ict.usc.edu/nld/qroup/  for  detailed  information  about 
the  group  and  its  projects).  The  Natural  Language  Dialogue  Group,  led  by  Dr.  David  Traum,  has 
developed  and  extensively  tested  characters  based  on  a  specialized  software  architecture  known  as 
Dialogue  ACT,  with  the  science  behind  character  interactions  rooted  in  cognitive  psychology,  personality 
theory,  and  linguistics,  among  other  disciplines.  Of  particular  appeal  to  the  CIR  project  team  is  the 
relevance  of  an  Army-funded  project  that  uses  a  Tactical  Questioning  system  (TACQ;  Gandhe,  DeVault, 
Roque,  et  al. ,  2008;  Roque  &  Traum,  2007;  Rushforth,  Gandhe,  Artstein,  et  al.,  2009;  Traum,  Leuski, 
Roque,  et  al.,  2008).  The  system  shows  promise  as  an  architecture  sophisticated  enough  to  address  the 
multifaceted  and  complex  modes  of  communication  essential  for  the  project’s  VP  dialogue  needs,  as  it 
allows  for  character  personality,  conditional  interactions,  an  affective  model,  and  social  (versus 
transactional)  interaction  (Rushforth,  Gandhe,  Artstein,  et  al.,  2009). 

While  the  CIR  team  worked  to  resolve  the  potential  VP-readiness  barriers  in  Quarter  4,  project  personnel 
at  ICT  continued  to  move  forward  with  other  dimensions  of  VP  development.  Among  other  project 
accomplishments,  ICT  worked  towards  system-readiness  in  the  following  areas:  finalized  the  VPTE  to 
allow  for  user  login  to  the  system  and  recording  of  user  data;  created  new  speech  recognition  system 
language  models  from  CIR  project  team  data;  and  integrated  speech  components  into  the  Unity  engine. 
They  also  performed  motion  capture  to  capture  a  set  of  animations  that  will  be  used  by  the  VP  characters, 
and  created  facial  (FACS)  animations  for  the  Castilla  character. 

VP  Testing  and  Refinement 

VP  testing  and  subsequent  refinements  are  integral  aspects  of  the  development  of  the  project  VP  (and 
eventually,  the  Virtual  Family  [VF]).  An  early  testing  session  was  conducted  with  12  MSW  student 
volunteers  over  two  days  at  the  start  of  Quarter  3.  User  feedback  was  used  by  ICT  to  further  develop  the 
character  dialogue. 

As  mentioned  in  the  previous  section,  a  second,  more  informal,  round  of  testing  was  conducted  to  assess 
the  system’s  readiness  for  classroom  implementation.  It  was  in  response  to  that  testing  that  the  project 
team  decided  a  new  dialogue  system  would  be  necessary  in  order  to  fulfill  the  VP  portion  of  the  project 
SOW. 

As  the  project  year  came  to  a  close,  ICT  personnel  were  making  plans  for  additional  developmental 
testing  with  MSW  student  volunteers  at  CIR,  scheduled  for  mid-October  201 1 .  As  the  new  software 
architecture  is  employed  and  the  dialogue  approaches  readiness  for  classroom  use,  additional  waves  of 
user  testing  will  be  conducted  in  order  to  provide  the  strongest  possible  training  tool  (i.e.,  the  most 
realistic  military  client-type  interaction  with  trainees). 

Preparations  for  Implementing  VP  as  a  Training  Tool 

In  addition  to  the  technological  and  training-focused  conceptual  development  necessary  for  VP 
implementation,  logistical  preparations  were  also  necessary.  During  the  first  half  of  Year  1,  the  necessary 
equipment  specifications  for  the  two  proposed  virtual  clinics  (i.e.,  the  Virtual  Patient  testing  and  training 
sites)  were  determined,  and  equipment  was  purchased  for  the  first.  Subsequently,  the  equipment  was 
installed  at  the  USC  City  Center  location,  where  CIR  and  the  project  team  are  located.  Equipment  and 
space  availability  for  the  Virtual  Clinic  at  the  USC  San  Diego  Academic  center  was  under  negotiation 
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early  in  the  project  year,  but  a  decision  was  subsequently  made  to  hold  off  on  purchasing  that  equipment 
until  such  time  as  VP-based  training  would  be  implemented  at  that  site. 


Barriers  to  Completion  of  Targeted  SOW  Accomplishments 

Throughout  Year  1,  we  believe  that  the  targeted  project  activities  and  incremental  SOW  accomplishments 
have  largely  proceeded  according  to  timeline.  The  notable  exception  to  this  is  the  ongoing  development 
of  our  first  VP,  slated  for  classroom  implementation  in  September  201 1 .  With  the  dialogue  testing 
completed  by  project  personnel  in  July  201 1  and  the  resulting  realization  that  the  existing  VP  dialogue 
engine  would  not  allow  for  the  necessary  complexity,  the  project  team  immediately  took  steps  to  resolve 
this  important  barrier.  As  a  result,  a  revised  plan  is  now  in  place,  employing  the  knowledge  and 
technology  of  ICT’s  Natural  Language  Dialogue  Group,  showing  promise  for  successful  completion  of  the 
first  VP  and  classroom/training  implementation  targeted  for  mid-  Year  2. 

Anticipating  the  targeted  SOW  accomplishments  for  project  Year  2,  we  foresee  some  additional  barriers, 
impacting  the  timely  and  effective  implementation  of  the  first  pilot  study  (examining  the  training  curriculum 
with  behavioral  health  providers),  originally  scheduled  for  completion  by  the  end  of  Phase  II  (March 
2012).  One  contributing  factor  is  the  online  CE  course  development.  In  the  original  plan,  the  online  CE 
courses  would  have  been  developed  concurrently  with  the  on-ground  courses,  implementing  course 
delivery  through  both  methods  around  the  same  time.  Given  the  resource-  and  time-intensive  nature  of 
developing  online  courses  (and  revising  them),  the  project  team  decided  to  first  develop,  implement  and 
evaluate  the  sequence  of  on-ground  courses  so  that  necessary  revisions  could  be  made  prior  to 
developing  the  online  versions.  While  allowing  for  higher-quality  online  courses,  the  decision  resulted  in 
the  tradeoff  of  a  delayed  timeline  for  online  course  rollout.  Further,  as  described  previously,  another 
contributing  factor  is  the  delay  in  VP  development  due  to  the  need  for  stronger  dialogue  architecture. 
Because  the  pilot  study  is  designed  to  examine  the  impact  of  the  complete  training  curriculum  (courses  + 
VP),  it  will  not  be  possible  to  execute  the  pilot  study  until  the  VP  is  ready  for  training  implementation.  A 
proposed  solution,  ensuring  the  ability  to  complete  a  rigorous,  high-quality  pilot  test,  will  be  discussed  in 
the  following  section  of  this  report. 


Recommended  Changes  to  Project  Timeline  and/or  Targeted  Accomplishments 

Considering  together  the  approved  SOW  and  project  plan,  project-related  successes  and 
accomplishments-to-date,  barriers,  and  lessons  learned,  there  are  a  few  key  areas  where  changes  to  the 
project  plan  would  serve  to  strengthen  the  quality  of  the  intervention  or  its  evaluation,  or  to  otherwise 
support  achievement  of  the  project’s  overarching  aims. 

Consolidation  of  the  First  and  Second  Pilot  Studies 

Original  project  plan.  The  original  project  plan  involved  conducting  two  distinct  pilot  studies  of 
the  (CE)  training  curriculum  with  advanced  behavioral  health  providers.  The  first  pilot  study,  slated  to 
occur  in  the  first  half  of  Year  2,  was  to  examine  the  impact  of  the  first  two  courses  ( Military  Culture  and 
Clinical  Practice  with  Servicemembers  and  Veterans)  along  with  training  via  the  Virtual  Patient,  among 
providers  working  with  individual  clients  (servicemembers  or  veterans);  the  second  pilot  study,  scheduled 
for  later  in  Phase  III ,  would  examine  the  impact  of  the  other  two  courses  (Families  Impacted  by  Military 
Service:  Understanding  and  Intervening  and  Health  Challenges  for  Wounded  Warriors  and  their 
Caregivers)  along  with  training  via  the  Virtual  Family,  among  providers  working  with  military  families. 
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Both  studies  would  examine  the  relative  impact  across  three  randomized  groups:  those  being  trained  on¬ 
ground;  those  being  trained  online;  and  those  not  receiving  the  training. 

Current  situation.  As  mentioned  previously,  the  online  CE  courses  have  not  yet  rolled-out,  due 
to  the  decision  to  develop  them  in  sequence  after  the  on-ground  CE  courses.  Further,  the  VP  is  not  yet 
ready  for  training  implementation.  Moving  forward  with  the  first  pilot  study  at  this  time  would  render  a 
comparison  of  on-ground  vs.  online  CE  training  impossible,  and  would  not  allow  for  examination  of  the 
impact  of  training  via  the  VP. 

Recommended  change.  The  recommended  solution  in  this  case,  allowing  for  a  rigorous, 
controlled  examination  of  the  full  training  curriculum,  is  to  consolidate  the  first  and  second  pilot  tests, 
conducting  them  together  at  the  time  the  second  pilot  is  slated  to  occur  in  Phase  III.  The  two  pilot  studies 
would  essentially  be  combined,  allowing  for  a  larger  sample  size  than  either  pilot  separately,  and  an 
examination  of  the  training  curriculum’s  impact  on  clinicians  and  their  work  with  military-related  clients  - 
either  individual  servicemembers  or  veterans  -  or  military  families.  No  benefit  would  be  lost  by  this 
consolidation,  which  was  originally  planned  as  two  separate  pilots  mostly  because  the  courses  were 
slated  to  be  phased-in  rather  than  being  released  together. 

Maintenance  of  Pi’s  Current  Time  Allocation  into  Years  2  and  3 

Original  project  plan.  Per  the  Cooperative  Agreement  budget  and  justification  documents,  Dr. 
Hassan,  Principal  Investigator,  would  devote  50%  effort  to  the  project  in  year  one,  60%  in  year  two,  and 
80%  in  year  three.  Thus,  there  would  be  an  increase  in  effort  over  time. 

Current  situation.  During  the  course  of  Year  1 ,  Dr.  Jeffrey  Wilkins,  a  CIR  Scientific  Advisory 
Group  member  and  former  VA  psychiatrist  with  expertise  in  substance  abuse  and  mental  health,  and  with 
a  lengthy  and  impressive  record  of  both  scholarship  and  leadership,  began  working  as  a  consultant  to 
CIR.  (Please  refer  to  Appendix  K  for  Dr.  Wilkins’  curriculum  vitae.)  As  Dr.  Hassan  began  to  realize  the 
constraints  on  his  own  time,  and  Dr.  Nissly  needed  to  ease  her  leadership  role  on  the  project  due  to  a 
geographic  relocation,  it  became  apparent  that  Dr.  Wilkins  would  be  uniquely  posed  to  work  closely  with 
Dr.  Hassan  and  assume  operational  leadership  of  the  project. 

Recommended  change.  The  recommendation  in  this  case  is  to  not  make  the  planned  effort 
increase  for  Dr.  Hassan,  but  rather  to  maintain  his  Year  1  effort  level  through  the  duration  of  the  project. 
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KEY  RESEARCH  ACCOMPLISHMENTS: 


■  Development  and  production  of  four  full  academic  online  military  social  work  MSW  student 
courses 

■  Rollout  of  first  two  online  military  social  work  MSW  student  courses,  complete  with  multimedia 
content 

■  Development  and  rollout  of  four  brief,  on-ground  continuing  education  courses  for  advanced 
behavioral  health  providers 

■  Completion  of  the  initial  evaluation  of  continuing  education  course  sequence 

■  Development  of  a  culturally-relevant  clinical  skill  assessment  system  utilizing  an  objective, 
structured  video  examination 

■  Implementation  of  a  large-scale,  comparison-group  study  of  the  MSW-level  military  social  work 
curriculum 

■  Solidification  of  first  Virtual  Patient  training  character,  with  accompanying  skill  development 
objectives,  character  background  and  clinical  presentation,  emotional  dimensions, 

■  Development  of  multiple  technological  dimensions  of  first  VP  training  character,  including 
graphics  of  physical  appearance  and  clinical  setting,  gestures  and  movements,  enhanced  speech 
recognition,  and  ability  to  record  user  data. 

■  Identification  of  software  architecture  with  capacity  to  deliver  the  complexity  of  dialogue 
necessary  for  training  behavioral  health  providers  in  non-linear  clinical  interactions 
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REPORTABLE  OUTCOMES: 


Courses  Developed 

■  Military  Culture  (1  academic  credit,  online  MSW  course) 

■  Clinical  Practice  with  Servicemembers  and  Veterans  (3  academic  credits,  online  MSW  course) 

■  Military  Culture  (7  hour,  on-ground  CE  course) 

■  Clinical  Practice  with  Servicemembers  and  Veterans  (15  hour,  on-ground  CE  course) 

■  Families  Impacted  by  Military  Service:  Understanding  and  Inten/ening  (15  hour,  on-ground  CE 
course) 

■  Health  Challenges  for  Wounded  Warriors  and  Their  Caregivers  (15  hour,  on-ground  CE  course) 

Courses  under  Development 

■  Families  Impacted  by  Military  Service:  Understanding  and  Intervening  (3  academic  credits,  online 
MSW  course) 

■  Health  Challenges  for  Wounded  Warriors  and  Their  Families  (3  academic  credits,  online  MSW 
course) 

■  Military  Culture  (6  hour,  online  CE  course) 

■  Clinical  Practice  with  Servicemembers,  Veterans,  and  Military  Families  (6  hour,  online  CE  course) 

■  Clinical  Skill  Building  Series  (6  hour,  online  CE  courses)  -  Intensive  clinical  skill  courses  that 
provide  in-depth  training  on  a  range  of  specific  therapeutic  models.  Topics  include: 

o  Prolonged  Exposure  Therapy  (PE):  PTSD 
o  Cognitive  Processing  Therapy  (CPT):  PTSD 
o  Cognitive  Behavioral  Therapy  (CBT):  Depression/anxiety 
o  Problem  Solving  Therapy  (PST) 
o  Solution  Focused  Therapy  (SFT) 
o  Motivational  interviewing  (Ml) 

Behavioral  Health  Providers  Trained 

■  26  online  MSW  students  in  courses  beginning  July  201 1 

■  59  on-ground  MSW  students  in  Fall  201 1  courses 

■  50  unique  behavioral  health  providers  trained  (10  having  completed  45+  hours  of  training) 

Manuscripts  in  Preparation 

■  Needs  assessment  of  civilian  behavioral  healthcare  providers  serving  veterans  and  military 
families. 

■  Application  of  Kirkpatrick’s  four-level  model  of  training  on  continuing  education  for  professional 
behavioral  health  providers. 

Presentations 

■  Hassan,  A.  (2011,  Sept).  Virtual  Patient.  TATRC  Telemedicine  Conference,  Anchorage,  AK. 

■  Hassan,  A.  (201 1 ,  June).  Veteran  Unemployment  Special  Interest  Working  Group.  Clinton  Global 
Initiative,  Chicago,  IL. 

■  Hassan,  A.  (2011,  June).  Reintegration  Partnership  Project.  Kings  College,  London,  U.K. 
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■  Hassan,  A.  (201 1 ,  May).  Military  Families.  Canadian  Embassy  Health  Research  Forum, 
Washington,  D.C. 

■  Hassan,  A.  (201 1 ,  May).  Center  for  Innovation  and  Research  on  Veterans  and  Military  Families. 
Canadian  Embassy  Health  Research  Forum,  Washington,  D.C. 

■  Nissly,  J.,  Williams,  J.,  &  Mor-Barak,  T.  (201 1 ,  May).  What’s  Next?  Avatars  for  Clinical  Training 
and  Interventions.  41st  National  Council  Conference  on  Mental  Health  and  Addictions.  San 
Diego,  CA. 

■  CIR  Project  Team.  (201 1 ,  April).  VP  demo  presentation  for  Congresswoman  Grace  Napolitano's 
staffers. 

■  Hassan,  A.  (201 1 ,  April).  Evidence-based  Practice  Implementation:  A  Toolkit  for  Managers. 
Network  for  Social  Work  Managers,  Baltimore,  MD. 

■  ICT  Virtual  Patient  Team.  (201 1 ,  March).  Game  Tech,  Orlando,  FL. 

■  ICT  Virtual  Patient  Team.  (201 1 ,  February).  Game  Developer’s  Conference,  San  Francisco,  CA. 

■  Hassan,  A.  (2011,  February).  Creating  a  Social  Work  Response  to  Veterans:  An  Urgent 
Challenge  in  the  Classroom.  Bachelors  Program  Directors  Conference,  Cincinnati,  OH. 

■  ICT  Virtual  Patient  Team.  (201 1 ,  January).  Consumer  Electronics  Show,  Las  Vegas,  NV. 

■  Hassan,  A.  (201 1 ,  January).  Chair:  Intervention  Science  and  Military  Family  Support.  Society  for 
Social  Work  Research  Annual  Conference,  Tampa,  FL 

■  Hassan,  A.  (2011,  January).  Panel  Symposium:  So  you're  Interested  in  Doing  Research  with 
Service  Members,  Veterans,  or  Military  Families?  Society  for  Social  Work  Research  Annual 
Conference,  Tampa,  FL. 

■  Nissly,  J.  and  Hassan,  A.  (2010,  November).  The  Virtual  Patient  Training  Environment:  A 
Revolutionary  Approach  to  Preparing  Social  Workers.  Uniformed  Services  Social  Worker 
Conference,  Phoenix,  AZ. 

■  Hassan,  A.  (2010,  November).  What  is  a  military  social  worker?  Uniformed  Services  Social 
Worker  Conference,  Phoenix,  AZ. 

■  Hassan,  A.,  Wooten,  N.,  Smith-Osborne,  A.,  &  Sable,  M.  (2010,  October).  Social  Work  Education 
Initiatives  for  Practice  with  Military  Personnel  and  their  Families.  Council  on  Social  Work 
Education  Annual  Program  Meeting,  Portland,  Oregon. 

■  Hassan,  A.,  Nissly,  J.,  &  Kim,  A.  (2010,  October).  Virtual  Patient  Training  Environment:  A 
Revolutionary  Approach  to  Preparing  Military  Social  Workers.  Council  on  Social  Work  Education 
Annual  Program  Meeting,  Portland,  Oregon. 

■  Hassan,  A.,  Black,  P.,  Matthieu,  M.,  &  Daley,  J.  (2010,  October).  Advanced  Social  Work  Practice 
with  Military,  Veterans,  and  their  Families.  Council  on  Social  Work  Education  Annual  Program 
Meeting,  Portland,  Oregon. 

■  Hassan,  A.  (2010,  October).  A  rapid  and  revolutionary  response  to  training  behavioral  health 
providers.  University  of  Missouri  Summit,  Columbia,  MO. 

■  Institute  of  Medicine  (IOM)  Panel  Member;  Substance  Use  Disorders:  Military  Service  Members 
and  Veterans  (2011) 

■  Institute  of  Medicine  (IOM)  Panel  Member;  Quality  Assurance  Program  for  Mental  Health 
Providers  and  Other  Health  Providers  (2010) 

Marketing  and  Publicity 

■  VP  Progress  Video,  for  use  in  presentations  and  to  create  awareness  of  the  VPTE  and  its 
applications.  Viewable  at  http://www.youtube.com/watch?v=2QIE7PeAYoc. 
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CONCLUSION: 


Substantial  progress  was  made  in  Year  1  with  regard  to  fulfilling  the  Phase  II  SOW.  Two  of  the  online 
graduate  (MSW)  courses  were  produced  and  launched,  with  the  remaining  courses  well  into  the 
production  phase.  Four  brief  on-ground  CE  courses  were  developed,  delivered  and  evaluated, 
demonstrating  very  positive  findings  in  the  realms  of  trainee  satisfaction,  learning,  and  intent  to  apply  the 
course  content  in  clinical  practice.  The  online  brief  CE  courses  are  also  being  produced,  with  the  first 
course  in  a  series  slated  for  online  launch  later  this  calendar  year.  A  quasi-experimental  comparison 
group  test  of  the  MSW  curriculum  was  begun  in  Quarter  4,  with  81  students  enrolled  into  the  study.  With 
student  knowledge,  skill,  sense  of  competence  and  curriculum  coherence  as  the  outcomes  of  interest,  the 
study  will  provide  data  towards  answering  whether  the  curriculum  is  effective  in  producing 
knowledgeable,  skilled  military  social  workers,  and  whether  virtual  curriculum  delivery  is  as  effective  as 
the  on-ground  method. 

Though  development  of  the  VP  did  not  proceed  as  quickly  in  Year  1  as  had  been  anticipated,  valuable 
lessons  were  learned  that  resulted  in  some  adjustments  in  personnel  and  approach  to  development. 
Character  graphics,  movements,  voice  recognition,  and  other  core  functions  were  completed  and  are 
undergoing  testing.  With  a  new  system  architecture  being  employed,  the  potential  for  creating  complex 
and  meaningful  character  dialogue  will  be  realized  in  a  way  that  it  could  not  have  been  previously. 

Even  after  the  considerable  time  needed  for  development  of  the  military  social  work/behavioral  health 
curriculum,  135  MSW  students  and  advanced  behavioral  health  providers  completed  the  courses  during 
the  first  year  of  the  project.  With  the  launch  of  the  online  courses  and  the  implementation  of  the  VP  for 
training  purposes,  the  reach  of  the  training  curriculum  will  be  extended  dramatically  in  Year  2.  Once  the 
online  curriculum  and  VP  have  been  fully  implemented,  a  randomized  study  will  be  needed  to  examine 
the  relative  outcomes  of  CIR  military  behavioral  health  training,  and  of  virtual  training  more  specifically. 
For  this  reason,  the  initial  pilot  study  planned  for  the  early  portion  of  Year  2  should  be  postponed  and 
consolidated  with  the  second  planned  pilot  study,  slated  for  late  2012. 
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Military  Social  Work  Subconcentration  Questionnaire 

As  a  new  graduate  of  the  School's  Military  Social  Work  Subconcentration,  we  would  greatly  appreciate  your  feedback 
regarding  the  military  social  work  curriculum  and  its  value  to  you  as  a  social  worker.  As  a  token  of  our  appreciation  for 
your  taking  the  time  to  complete  this  questionnaire,  it  will  be  our  pleasure  to  send  an  iTunes  gift  card  to  you. 

Please  keep  the  following  description  of  the  MSW  program  subconcentration  in  Military  Social  Work  and  Veterans 
Services  in  mind  as  you  respond  to  the  subsequent  questions: 

Military  Social  Work  and  Veterans  Services 

This  specialized  area  of  study  prepares  individuals  to  provide  a  full  range  of  human  services  to  the  nation's  military 
personnel,  veterans  and  their  families,  helping  them  cope  with  the  stresses  of  military  life,  including  managing 
prolonged  deployments  and  transitioning  back  into  a  home  environment. 


***  We  would  be  grateful  for  any  specific  comments  you  provide,  particularly  regarding  items  marked  as  strongly 
disagree  or  strongly  agree.  ***  Please  feel  free  to  write  in  the  margins. 


SOWK  642:  Military  Culture  as  a  Workplace  Environment  (Please  check  here  □  if  you  did  not  take  the  Military  Culture 
course,  and  skip  to  the  following  section.) 

1.  The  breadth  of  the  material  covered  in  the  Military  Culture  course  was  appropriate. 


□l 

□2 

□3 

□4 

□5 

Strongly  disagree 

Disagree 

Neither  agree 
nor  disagree 

Agree 

Strongly  agree 

If  you  selected  Strongly  disagree  or  disagree,  how  so  (i.e.,  too  broad,  not  broad  enough)? 


2.  The  depth  of  material  covered  in  the  Military  Culture  course  was  appropriate. 


□  1 

□2 

□3 

□4 

□5 

Strongly  disagree 

Disagree 

Neither  agree 
nor  disagree 

Agree 

Strongly  agree 

If  you  selected  Strongly  disagree  or  disagree,  how  so  (i.e.,  too  deep,  not  deep  enough)? 


3.  The  Military  Culture  curriculum  was  poorly  organized  and  did  not  flow  in  a  logical  sequence. 


□  1 

□2 

□3 

□4 

□5 

Strongly  disagree 

Disagree 

Neither  agree 
nor  disagree 

Agree 

Strongly  agree 
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4.  My  approach  to  practice  with  military-related  clients  was  impacted  by  taking  the  Military  Culture  course. 

Ol  O2  EH  ;  EH  ;  I  I 

Strongly  disagree  Disagree  Neither  agree  Agree  Strongly  agree 

nor  disagree 


In  what  way(s)? 


5.  My  level  of  professional  competence  was  improved  as  a  result  of  my  participation  in  the  Military  Culture 


course. 

□1 

□2 

□3 

□4 

□5 

Strongly  disagree 

Disagree 

Neither  agree 

Agree 

Strongly  agree 

nor  disagree 

SOWK  641:  Treating  Trauma  and  Post  Traumatic  Stress  (Please  check  here  □  if  you  did  not  take  the  Treating  Trauma 
course,  and  skip  to  the  following  section.) 

6.  The  breadth  of  the  material  covered  in  the  Treating  Trauma  course  was  appropriate. 


□  1 

□2 

□3 

□4 

□5 

Strongly  disagree 

Disagree 

Neither  agree 
nor  disagree 

Agree 

Strongly  agree 

If  you  selected  Strongly  disagree  or  disagree,  how  so  (i.e.,  too  broad,  not  broad  enough)? 


7.  The  depth  of  material  covered  in  the  Treating  Trauma  course  was  appropriate. 


□  1 

□2 

□3 

□4 

□5 

Strongly  disagree 

Disagree 

Neither  agree 
nor  disagree 

Agree 

Strongly  agree 

If  you  selected  Strongly  disagree  or  disagree,  how  so  (i.e.,  too  deep,  not  deep  enough)? 


8.  The  Treating  Trauma  curriculum  was  poorly  organized  and  did  not  flow  in  a  logical  sequence. 

□1  m2  O3  D4  D5 

Strongly  disagree  Disagree  Neither  agree  Agree  Strongly  agree 

nor  disagree 

9.  My  approach  to  practice  with  military-related  clients  was  impacted  by  taking  the  Treating  Trauma  course. 

□1  EH2  IHI3  CI4  CI5 

Strongly  disagree  Disagree  Neither  agree  Agree  Strongly  agree 

nor  disagree 

In  what  way(s)? 
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10.  My  level  of  professional  competence  was  improved  as  a  result  of  my  participation  in  the  Treating  Trauma 


course. 

□1 

□2 

□3 

□4 

□5 

Strongly  disagree 

Disagree 

Neither  agree 

Agree 

Strongly  agree 

nor  disagree 

SOWK  640:  Clinical  Practice  with  the  Military  Family  (Please  check  here  □  if  you  did  not  take  the  Military  Family 
course,  and  skip  to  the  following  section.) 

11.  The  breadth  of  the  material  covered  in  the  Military  Family  course  was  appropriate. 


□  1 

□2 

□3 

□4 

□5 

Strongly  disagree 

Disagree 

Neither  agree 
nor  disagree 

Agree 

Strongly  agree 

If  you  selected  Strongly  disagree  or  disagree,  how  so  (i.e.,  too  broad,  not  broad  enough)? 


12.  The  depth  of  material  covered  in  the  Military  Family  course  was  appropriate. 


□  1 

□2 

□3 

□4 

□5 

Strongly  disagree 

Disagree 

Neither  agree 
nor  disagree 

Agree 

Strongly  agree 

If  you  selected  Strongly  disagree  or  disagree,  how  so  (i.e.,  too  deep,  not  deep  enough)? 


13.  The  Military  Family  curriculum  was  poorly  organized  and  did  not  flow  in  a  logical  sequence. 

□i  u2  m3  m4  n5 

Strongly  disagree  Disagree  Neither  agree  Agree  Strongly  agree 

nor  disagree 


14.  My  approach  to  practice  with  military-related  clients 

□i  n2  n3 

Strongly  disagree  Disagree  Neither  agree 

nor  disagree 


was  impacted  by  taking  the  Military  Family  course. 

□4  Ds 

Agree  Strongly  agree 


In  what  way(s)? 


15.  My  level  of  professional  competence  was  improved  as  a  result  of  my  participation  in  the  Military  Family  course. 


□1 

□2 

□3 

□4 

□5 

Strongly  disagree 

Disagree 

Neither  agree 
nor  disagree 

Agree 

Strongly  agree 
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16.  My  approach  to  practice  with  military-related  clients  was  impacted  by  having  a  military-focused  field 


experience. 

□  1 

□2 

□3 

□4 

□5 

Strongly  disagree 

Disagree 

Neither  agree 

Agree 

Strongly  agree 

nor  disagree 

In  what  way(s)? 


17.  My  level  of  professional  competence  was  improved  as  a  result  of  my  military  social  work  field  experience. 

O2  □ ;  □ :  I  I 

Strongly  disagree  Disagree  Neither  agree  Agree  Strongly  agree 

nor  disagree 

What  sources  of  information  lead  you  to  this  conclusion  (i.e.,  how  do  you  know)?: 


18.  Considered  in  its  entirety,  the  subconcentration  curriculum  provided  the  right  focus  and  level  of 

education/training  necessary  for  preparing  a  new  social  worker  to  work  effectively  with  military-related 


populations. 

□1 

□2 

□3 

□4 

□5 

Strongly  disagree 

Disagree 

Neither  agree 

Agree 

Strongly  agree 

nor  disagree 

19.  The  individual  course  content  was  overly  redundant  and/or  not  well-coordinated. 

□1  o2  n3  n4  m5 

Strongly  disagree  Disagree  Neither  agree  Agree  Strongly  agree 

nor  disagree 

In  what  way(s)? 


20.  In  retrospect,  what  topics  would  you  like  to  have  seen  covered,  that  weren't?  (...or  weren't  covered  in  sufficient 
depth)? 
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21.  As  a  result  of  my  involvement  in  the  subconcentration,  I  will  be  confident  in  seeking  work  opportunities  with 
servicemembers,  veterans  and  military  families. 

Ol  d2  EH  ;  EH  ;  I  I 

Strongly  disagree  Disagree  Neither  agree  Agree  Strongly  agree 

nor  disagree 

22.  In  one  sentence  or  phrase,  what  is  the  single  most  important  thing  you  gained  through  your  involvement  in  the 
subconcentration? 


23.  I  would  recommend  the  subconcentration  to  a  friend  or  student  colleague  interested  in  working  with  veterans 


and  military  families. 

□ 

□ 

□3 

□4 

□5 

Strongly  disagree  Disagree 

Neither  agree 
nor  disagree 

Agree 

Strongly  agree 

So  that  we  might  better  understand  the  context  of  the  feedback  we  receive,  please  respond  to  the  following 
demographic  items. 

Service  History: 

Do  you  presently,  or  have  you  in  the  past,  served  in  one  or  more  of  the  U.S.  Uniformed  Services? 

□1  No  n2Yes . Air  Force  D2Army  D3  Coast  Guard 

□4  Marine  Corps  D5  Navy  n6  Other: _ 

Age:  n±  20-29  d2  30-39  D3  40-49  n450+ 

Gender:  Female  D2  Male 


Ethnicity:  Di  Asian/Pacific  Islander 

□4  Native  American 

□2  Black/African-American 
□5  Anglo/White 

□3  Hispanic/Latino(a) 
□fi  Other 

Concentration:  diChildren  &  Families 
□4  Mental  Health 

□2COPA 
□5  Work  and  Life 

□3  Health 

T hank  you  for  taking  the  time  to  respond  to  our  questions.  The  information  you  provided  will  be  invaluable 
in  further  enhancing  the  content  and  quality  of  the  subconcentration  and  its  curriculum.  We  value  your 
commitment  to  serving  our  nation's  servicemembers  and  their  families,  and  wish  you  the  best  in  your  career! 


Appendix  B 

CEU  Course  Participant  Feedback  Questionnaire  (Military  Culture  Version) 
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CEU  Course  Participant  Feedback  Questionnaire 


In  order  to  provide  the  highest  possible  quality  of  professional  military  behavioral  health  training,  we 
would  value  your  perspectives  on  the  following.  Please  respond  to  each  using  the  following  rating 
system: 

1  =  Strongly  disagree  /  2  =  Disagree  /  3  =  Neither  agree  nor  disagree  /  4  =  Agree  /  5  =  Strongly  agree 

We  would  be  grateful  for  any  specific  comments  you  provide,  particularly  regarding  items  marked  as 
strongly  disagree  or  strongly  agree.  (Please  feel  free  to  write  in  the  margins.) 


1.  The  instructors  were  interesting  and  kept  my  attention  throughout  the  training. 
Ol  O2  ED  ;  □  ;  I  I 

Strongly  disagree  Disagree  Neither  agree  Agree  Strongly  agree 

nor  disagree 


The  material  was  clearly  presented. 

Ol  O2  EH  ; 

Strongly  disagree  Disagree  Neither  agree 

nor  disagree 


□4  Ds 

Agree  Strongly  agree 


3.  The  balance  of  instruction  methods  (lecture,  discussion,  small  group  work)  was  about  right. 


□1 

Strongly  disagree 


□  2 

Disagree 


□3 

Neither  agree 
nor  disagree 


The  activities  and  exercises  aided  in  my  learning. 

□i  n2  m3 

Strongly  disagree  Disagree  Neither  agree 

nor  disagree 


□4 

Agree 


□5 

Strongly  agree 


□4  n5 

Agree  Strongly  agree 


5.  The  breadth  of  the  material  covered  was  appropriate. 

Ol  O2  EH  ; 

Strongly  disagree  Disagree  Neither  agree 

nor  disagree 

6.  The  depth  of  material  covered  was  appropriate. 

Ol  O2  EH  ; 

Strongly  disagree  Disagree  Neither  agree 

nor  disagree 


□4 

Agree 


□5 

Strongly  agree 


□4  Ds 

Agree  Strongly  agree 


This  course  was  delivered  at  an  appropriate  level  for  practitioners  with  my  level  of  experience. 

Ol  O2  EH  ;  EH  ;  I  I 

Strongly  disagree  Disagree  Neither  agree  Agree  Strongly  agree 

nor  disagree 
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8.  It  felt  like  there  was  a  disconnect  between  the  instructors'  perspectives  (having  served)  and 


mine,  as  a  civilian  provider. 

□  l  O2 

□3 

□4 

□5 

□9 

Strongly  disagree  Disagree 

Neither  agree 

Agree 

Strongly  agree 

N/A 

nor  disagree 

(1  served,  too) 

9.  This  course  made  me  aware  of  the  differences  in  civilian  and  military  culture  with  regard  to 

behavioral  health  practice  issues. 

Ol  O2  EH  ;  EH  ;  I  I 

Strongly  disagree  Disagree  Neither  agree  Agree  Strongly  agree 

nor  disagree 

10.  As  a  result  of  what  I  learned  in  this  course,  I  can  think  of  two  concrete  strategies  I  can  use  in  my 
practice  with  veteran  clients. 

Ol  O2  EH  ;  EH  ;  I  I 

Strongly  disagree  Disagree  Neither  agree  Agree  Strongly  agree 

nor  disagree 

11.  Participation  in  this  course  will  impact  how  I  interact  with  my  military-related  clients. 

Ol  O2  EH  ;  EH  ;  I  I 

Strongly  disagree  Disagree  Neither  agree  Agree  Strongly  agree 

nor  disagree 

12.  As  a  result  of  this  training,  I  will  be  more  confident  in  seeking  out  opportunities  to  work  with 
veterans  and  military  families  in  my  practice. 

□1  O  2  D3  m4  d5 

Strongly  disagree  Disagree  Neither  agree  Agree  Strongly  agree 

nor  disagree 

13.  Today's  training  made  me  feel  less  equipped  to  practice  effectively  with  veterans  and  military 


families. 

□1 

□2 

□3 

□4 

□5 

Strongly  disagree 

Disagree 

Neither  agree 

Agree 

Strongly  agree 

nor  disagree 

14.  After  this  course,  I  am  confident  about  where  to  look  for  additional  resources  and  information 
on  working  with  veteran  and  military  family  clients. 

d  1  EH2  CI3  IZI4  IZI5 

Strongly  disagree  Disagree  Neither  agree  Agree  Strongly  agree 

nor  disagree 

15.  In  a  sentence  or  two,  what,  if  any,  barriers  do  you  anticipate  could  prevent  you  from  using  the 
knowledge  you  acquired  today  in  your  practice? 
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16.  In  one  sentence  or  phrase,  what  is  the  single  most  important  thing  you  learned  in  this  training? 


17.  The  course  covered  the  right  amount  of  information  for  the  length  of  the  training. 

Ol  O2  EH  ;  EH  ;  I  I 

Strongly  disagree  Disagree  Neither  agree  Agree  Strongly  agree 

nor  disagree 


18.  The  topics  covered  were  what  I  was  hoping  to  see  in  this  course. 

Ol  O2  EH  ;  EH  ; 

Strongly  disagree  Disagree  Neither  agree  Agree 

nor  disagree 


□5 

Strongly  agree 


19.  Please  describe  any  topics  that  you  expected  to  be  covered,  but  weren't. 


20.  Please  describe  any  areas  in  which  you  felt  more  attention  should  have  been  provided  to  the 
material  (within  the  course  time  constraints). 


21.  Overall,  the  course  was  valuable. 

□1  \32  EH  3  ED4  ED5 

Strongly  disagree  Disagree  Neither  agree  Agree  Strongly  agree 

nor  disagree 

22.  A  course  like  this  should  be  compulsory  for  practitioners  working  with  veterans  and  military 


families. 

□1 

□2 

□3 

□4 

□5 

Strongly  disagree 

Disagree 

Neither  agree 

Agree 

Strongly  agree 

nor  disagree 

23.  I  would  recommend  this  course  to  a  friend  or  colleague  interested  in  working  with  veterans 


and  military  families. 

□ 

□ 

□3 

□4 

□5 

Strongly  disagree  Disagree 

Neither  agree 
nor  disagree 

Agree 

Strongly  agree 

24.  I  would  consider  $ _ : _ to  be  a  suitable  cost  for  this  course  as  a  Continuing  Education 


course. 
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So  that  we  might  better  understand  the  context  of  the  feedback  we  receive,  please  respond  to  the 
following  demographic  items. 

Do  you  presently,  or  have  you  in  the  past,  served  in  one  or  more  of  the  U.S.  Uniformed  Services? 


□i  No 

□  2Yes  [Which 

branch(es)?: 

1 

Age: 

Under  25 

□2  25-34 

□  3  35-44  □„  45-54 

□5  55-64 

□  6  65  + 

Gender: 

□  1  Female 

□2  Male 

□3  Transgender 

Ethnicity: 

□  1  Asian/Pacific  Islander 
□4  Native  American 

□2  Black/African-American 
□5  Anglo/White 

□3  Hispanic/Latino(a) 

□fi  Other 

Professional  discipline:  Di  Marriage  &  Family  Therapist  D2  Nurse  D3  Psychologist 

□4  Social  Worker  D5 Other _ 


Licensed  as  a  behavioral  health  provider:  Yes  D2  In  process  (registered)  IU3  No 

Years  post-graduate  practice:  □1<5  D2  5-9  D3 10-14  D4 15-19  CDs  20+ 

Current  employment  status:  Di  Full-time  behavioral  health  D2  Part-time  behavioral  health 

□3  Employed,  non-behavioral  health  D4  Not  currently  employed 

Current  practice  setting(s):  diAging  IU2 Child  Welfare  IH3Flealth 

□4  Mental  Health  D5 Substance  Abuse  D6  Other _ 

What  portion  of  your  work  is  spent  in  direct  practice  with  individuals,  families  or  small  groups? 

□1  All  or  nearly  all  D2  Most  (51-90%)  D3 Some  (11-50%)  D4  Little  (<10%) 

What  portion  of  your  work  is  with  military-related  populations  (i.e.,  servicemembers,  veterans  or 
military  families)? 

□1  All  or  nearly  all  D2  Most  (51-90%)  D3 Some  (11-50%)  D4  Little  (<10%) 


T hank  you  for  taking  the  time  to  respond  to  our  questions.  The  information  you  provided  will  be  invaluable 
in  further  enhancing  the  content  and  quality  of  this  course.  We  value  your  commitment  to  serving  our 
nation's  servicemembers  and  their  families,  and  hope  that  you  have  gained  valuable  information  that  you 
will  apply  in  their  service. 


Appendix  C 

Military  Culture  Knowledge  Pretest 
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Military  Culture  Knowledge  Pretest 

Please  select  the  best  answer  for  each  of  the  following  questions. 

1.  What  do  Army  servicemembers  shout  as  a  display  of  their  camaraderie  and  pride? 

a.  Ooh-Rah! 

b.  Semper  Fi! 

c.  Hooah! 

d.  Army  Strong! 

2.  What  does  MOS  stand  for? 

a.  Military  Occupational  Specialty 

b.  Mission  Operational  Strategy 

c.  Munitions  Operating  Sergeant 

d.  Marine  Officer  School 

3.  Which  of  the  following  is  NOT  a  description  of  most  National  Guard  and  Reserve  Component 
members? 

a.  Unless  activated,  they  typically  serve  one  weekend  each  month  at  an  installation  near  their 
homes 

b.  They're  frequently  deployed  individually,  and  not  necessarily  as  an  entire  unit 

c.  Their  communities  are  largely  comprised  of  other  military  families 

d.  By  law,  their  "day  jobs"  must  be  held  for  them  during  active  duty  service  periods 

4.  "Soldier"  is  a  term  used  to  describe... 

a.  an  infantry  unit  member 

b.  a  member  of  the  Marine  Corps 

c.  a  member  of  the  Army 

d.  any  member  of  the  U.S.  armed  forces 

5.  Which  of  the  following  is  NOT  an  issue  likely  to  be  encountered  by  children  in  military  families? 

a.  Dealing  with  long-term  separation  from  one  or  both  parents 

b.  Attending  school  with  classmates  who  don't  understand  military  family  life 

c.  PCSing 

d.  Lacking  social  or  recreational  activities 

6.  Please  select  the  response  that  lists  servicemembers  from  lowest  to  highest  rank: 

a.  Enlistee,  Commissioned  Officer,  Warrant  Officer 

b.  Warrant  Officer,  Non-Commissioned  Officer,  Commissioned  Officer 

c.  Enlistee,  Recruit,  Officer 

d.  Enlistee,  Warrant  Officer,  Commissioned  Officer 
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7.  Rate  is  a  term  used  to  describe  the  servicemembers'  job  specialty  in  which  branch  of  service: 

a.  Army 

b.  Marine  Corps 

c.  Navy 

d.  Air  Force 

8.  Which  health  insurance  company  serves  Military  Families  with  health  benefits  options? 

a.  Flumana 

b.  Blue  Cross 

c.  TRICARE 

d.  Liberty  Mutual 

9.  Which  branch  of  service  falls  under  the  US  Navy? 

a.  Army 

b.  Coast  Guard 

c.  Marine  Corps 

d.  Air  Force 

10.  All  service  members  are  required  to  live  in  on-base  housing  for  their  first  year  of  active  duty 
service. 

a.  True 

b.  False 

11.  ACUs  are  worn  every  day  at  formation  to  show  respect  for  the  American  flag. 

a.  True 

b.  False 

12.  DEERS  stands  for: 

a.  Defense  Enlisted  Engineering  Register  System 

b.  Defense  Enrollment  Eligibility  Reporting  System 

c.  Dental  Enlisted  Enrollment  Reserve  Selection 

d.  Defense  Enlisted  Eligibility  Reporting  System 

13.  Can  an  active  duty  service  member  seek  mental  health  services  off-post/off-base  without  prior 
authorization? 

a.  True 

b.  False 

14.  Domestic  violence  is  not  a  mandatory  report  for  all  service  members. 

a.  True 

b.  False 

15.  What  organization  within  the  military  is  responsible  for  addressing  child  maltreatment? 

a.  NCIS 

b.  Family  Advocacy 

c.  Family  Readiness  Services 

d.  BX 
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16.  A  service  member  can  refer  to  his  or  her _ regarding  information  about  pay,  vacation  days  and 

allotments. 

a.  PEYS 

b.  LES 

c.  LPS 

d.  DDF 

17.  Active  duty  service  members  train  at  a  separate  boot  camp  than  the  reserve  service  members. 

a.  True 

b.  False 

18.  The  EFMP  is  a  mandatory  enrollment  program  that  works  with  other  military  and  civilian  agencies 
to  provide  comprehensive  and  coordinated  community  support,  housing,  educational,  medical,  and 
personnel  services  to  families  with  special  needs.  It  stands  for: 

a.  Emergency  Family  Mitigation  Program 

b.  Exceptional  Family  Member  Program 

c.  Enlisted  Family  Member  Program 

d.  Express  Family  Military  Program 

19.  Who  determines  the  uniform  of  the  day? 

a.  President  of  the  United  States 

b.  Joint  Chief  of  Staff 

c.  Base/Post  Commander 

d.  Unit  Commander 

20.  As  a  civilian  driving  on  post/base,  you  notice  cars  pulled  over  to  the  side,  the  national  anthem 
playing  and  everyone  in  uniform  is  saluting.  What  do  you  do? 

a.  Call  911  on  your  cell  phone 

b.  Pull  over  and  wait  for  normal  activity  to  resume 

c.  Hurry  up  and  get  off  the  post/base 

d.  Find  a  parking  lot  to  pull  over  and  find  shelter 


How  extensive  would  you  say  is  your  overall  knowledge  of  military  culture  and  the  life  of  a 
servicemember  or  military  family? 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

1 

Very 

Little 

2 

3 

4 

5 

6 

7 

Very 

Extensive 

Appendix  D 

Clinical  Practice  with  Servicemembers  and  Veterans  Knowledge  Pretest 
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Clinical  Practice  with  Servicemembers  &  Veterans 
Knowledge  Pretest 

Please  select  the  best  answer  for  each  of  the  following  questions. 

1.  "Dwell"  time  refers  to: 

a.  The  transition  between  serving  in  a  combat  zone  and  returning  home. 

b.  The  period  after  a  significant  life  stressor,  but  before  the  psychological  sequelae  manifest. 

c.  The  time  after  a  servicemember  experiences  a  traumatic  event  but  before  she  seeks 
treatment  from  a  professional. 

d.  The  period  a  servicemember  spends  at  home  or  on  a  military  base  between  deployments. 

2.  A  majority  of  servicemembers  returning  from  deployment  to  OEF  or  OIF  experience  symptoms 
of  PTSDorTBI. 

a.  True 

b.  False 

3.  Instead  of  fear,  patients  experiencing  PTSD  often  display  this  emotion  as  an  alternative. 

a.  Remorse 

b.  Guilt 

c.  Grief 

d.  Anger 

4.  Differential  diagnosis  of  PTSD  can  be  more  difficult  than  with  many  other  diagnoses. 

a.  True 

b.  False 

5.  Which  part  of  the  brain  is  involved  in  attaching  emotions  to  incoming  stimuli  and  responding  to 
fear? 

a.  parietal  lobe 

b.  hippocampus 

c.  amygdala 

d.  corpus  collosum 

6.  Energy  drinks  such  as  Red  Bull  and  Five  Hour  Energy  can  increase  symptoms  of  TBI  or  PTSD. 

a.  True 

b.  False 

7.  Which  of  the  following  is  the  "essential  ingredient"  of  PTSD? 

a.  flashbacks 

b.  disordered  thinking 

c.  traumatic  stressor 

d.  memory  disruption 
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8.  Per  VA  treatment  guidelines,  which  of  the  following  is  an  evidence-based  treatment  for  PTSD? 

a.  Stress  Inoculation  Training  (SIT) 

b.  Pharmacotherapy  via  use  of  SSRIs  (Zoloft  &  Paxil) 

c.  Exposure  Therapy 

d.  All  of  the  above 

9.  What  does  EMDR  stand  for? 

a.  Eye  Movement  Desensitization  &  Reprocessing 

b.  Early  Military  Defense  Response 

c.  Earleblum  Military  Depression  index  -  Revised 

d.  Extrasensory  Motor  Development  Resilience 

10.  Which  of  the  following  treatments  is  often  used  to  treat  recurring  nightmares? 

a.  Battlemind 

b.  SSRIs 

c.  Imagery  Rehearsal  Therapy 

d.  Prolonged  Exposure  therapy 

11.  There  are  at  least  two  evidence-based  therapies  that  have  been  demonstrated  to  be  effective  in 
eliminating  the  presence  of  significant  PTSD  symptoms. 

a.  True 

b.  False 

12.  Which  of  the  following  is  least  commonly  seen  in  veterans  with  PTSD? 

a.  Major  Depressive  Disorder 

b.  Thought  disorder 

c.  Substance  abuse 

d.  Parasuicidal  behavior 

13.  Which  of  the  following  is  often  a  very  stressful  situation? 

a.  Going  through  a  BRAC 

b.  Being  PCSed 

c.  Serving  as  an  individual  augmentee 

d.  All  of  the  above 

14.  Which  of  the  following  might  be  likely  to  trigger  a  flashback  among  a  combat  veteran  with 
PTSD? 

a.  Hearing  firecrackers  on  Chinese  New  Year 

b.  Stepping  outside  on  an  extremely  hot  day 

c.  Both  a  and  b 

d.  None  of  the  above 

15.  Which  of  the  following  is  not  a  phase  of  the  deployment  cycle? 

a.  Deployment 

b.  Return  from  Deployment/Redeployment 

c.  Discharge 

d.  Predeployment 
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16.  According  to  military  research  (MHAT-VI),  which  of  the  following  is  the  most  commonly- 
reported  combat-related  stressful/traumatic  event? 

a.  Receiving  artillery,  rocket  or  mortar  fire 

b.  Seeing  an  ill/injured  woman  or  child  one  couldn't  help 

c.  Having  a  close  call/taking  a  hit  but  being  saved  by  one's  gear 

d.  Handling  or  uncovering  human  remains 

17.  Which  of  the  following  Battlemind  skills  is  highly  valuable  down  range  but  likely  problematic  in 
civilian  life? 

a.  Targeted  Aggression 

b.  Tactical  Awareness 

c.  Non-Defensive  Driving 

d.  All  of  the  above 

18.  What  distinguishes  an  Acute  Stress  Reaction  from  Acute  Stress  Disorder  (ASD)? 

a.  The  elapsed  time  from  stressor/traumatic  event  to  onset  of  symptoms 

b.  The  duration  of  the  symptoms 

c.  The  severity  of  the  symptoms 

d.  The  nature  of  the  symptoms 

19.  Which  of  the  following  signs  or  symptoms  is  shared  by  PTSD  and  TBI? 

a.  Dizziness 

b.  Flashbacks 

c.  Headaches 

d.  Irritability 

20.  Per  VA  treatment  guidelines,  which  of  the  following  is  not  a  first-line  treatment  for  Major 
Depressive  Disorder? 

a.  Cognitive  Behavioral  Therapy  (CBT) 

b.  Interpersonal  Psychotherapy  (IPT) 

c.  Acceptance  and  Commitment  Therapy  (ACT) 

d.  Problem-Solving  Therapy  (PST) 


How  extensive  would  you  say  is  your  overall  knowledge  of  military  culture  and  the  life  of  a 
servicemember  or  military  family? 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

1 

Very 

Little 

2 

3 

4 

5 

6 

7 

Very 

Extensive 

Appendix  E 

Families  Impacted  by  Military  Service:  Understanding  and  Intervening  Knowledge  Pretest 
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Families  Impacted  by  Military  Service:  Understanding  and  Intervening 

Knowledge  Pretest 


Please  select  the  best  answer  for  each  of  the  following  questions. 


1.  The  Exceptional  Family  Member  Program  is  associated  with: 

a.  Gifted  children  of  service  personnel 

b.  "Special  needs"  dependents  of  service  personnel 

c.  Determining  servicemember  duty  station 

d.  Both  b  &  c 

2.  Comparing  military  culture  and  veteran  culture,  which  of  the  following  is  accurate? 

a.  Military  culture  emphasizes  mission  readiness  and  veteran  culture  emphasizes  benefit 
eligibility 

b.  Veteran  culture  prioritizes  fitness  for  duty  whereas  military  culture  accepts  all  ranges  of 
fitness  for  duty 

c.  Military  culture  and  veteran  culture  have  historically  had  a  superb  and  seamless 
transition  process  for  newly  released  military  personnel. 

d.  All  of  the  above 

3.  Due  to  their  developmental  stage,  adolescent  clients  may  be  more  likely  to  reveal  episodes  of 

family  violence  at  assessment  than  adult  clients. 

a.  True 

b.  False 

4.  According  to  Walsh,  key  processes  in  family  resilience  include: 

a.  Belief  systems 

b.  Organizational  patterns 

c.  Communication 

d.  All  of  the  above 

5.  During  the  sustainment  phase  of  the  deployment  cycle,  the  deployed  servicemember's  spouse: 

a.  Is  anticipating  servicemember's  departure 

b.  Develops  new  routines  during  servicemember's  absence 

c.  Is  in  shock 

d.  All  of  the  above 

6.  Family  Care  Plans  are  essential  for: 

a.  Military  single  parent  households 

b.  Dual-military  couple  families 

c.  Private  contractors 

d.  Both  a  &  b 
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7.  Sherman  et  al.'s  14-session  curriculum  of  monthly  workshops  for  caregivers  of  veterans  known 

as  the  Support  and  Family  Education  (SAFE)  Program,  does  all  of  the  following  EXCEPT: 

a.  Teach  caregivers  about  PTSD  and  other  mental  disorders 

b.  Reduce  stigma  associated  with  mental  illness 

c.  Provide  linkage  with  community  resources 

d.  Provide  individual  psychotherapy  for  family  caregivers 

8.  TAPS  stands  for: 

a.  Transitional  Accountability  Program  and  Stress 

b.  Temporary  Assistance  Protection  Services 

c.  Tragedy  Assistance  Program  for  Survivors 

d.  None  of  the  above 

9.  The  FOCUS  program  (Families  Overcoming  Under  Stress)  provides  Navy  and  Marine  families 

with: 

a.  Family  support  for  deployment  and  resilience  building 

b.  Cognitive  behavioral  interventions  for  stressed  families 

c.  A  comprehensive  soldier  fitness  program 

d.  None  of  the  above 

10.  According  to  the  Integrated  Behavioral  Couple  Therapy  (IBCT)  model,  in  addition  to  partners' 
actions,  which  of  the  following  is  an  additional  source  of  relationship  problems: 

a.  Conversational  tone 

b.  Emotional  reactivity  to  actions 

c.  Family  stressors 

d.  Intrusive  thoughts 

11.  Bowen's  model  of  community  support  suggests  which  key  factor  influences  military  family 
adaptation? 

a.  Financial  resources 

b.  Sense  of  community 

c.  Family  motivation  to  stay  in  the  military 

d.  Cognitive  coping  skills 

12.  Fluebner's  community  capacity-building  model  suggests  which  factor  can  impact  military 
family  outcomes? 

a.  Deployment  frequency 

b.  View  of  extended  family  on  military  service 

c.  Social  capital 

d.  Branch  of  service 

13.  According  to  Figley,  family  trauma  treatment  should  include: 

a.  Rebuilding  family  safety 

b.  Exposure  techniques  for  family  members 

c.  Re-establishing  family  structure 

d.  Both  a  &  d 
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14.  In  McCubbin's  Family  Stress  Model  (Double  ABC-X  Model),  the  A  stands  for: 

a.  Accumulation  of  stressful  events 

b.  Activating  events 

c.  Autonomous  response 

d.  Adaptation 

15.  Which  of  the  following  is  a  stage  in  the  military  life  cycle? 

a.  Trajectory  events  unique  to  the  individual 

b.  Military  culture  adaptation 

c.  Family  reaction  to  military  career  decisions 

d.  Career  identity  reintegration 

16.  Daley's  view  of  military  ethnicity  emphasizes  what  key  features? 

a.  Mission  is  first  priority 

b.  Flexibility  is  important  in  social  conventions 

c.  Military  prioritizes  civilian  resources  to  take  care  of  military  families  rather  than  a  sense 
of  taking  care  of  their  own 

d.  All  of  the  above 

17.  Which  of  the  following  accurately  reflects  Pat  Conroy's  experience  growing  up  in  a  military 
family? 

a.  The  military  is  a  very  visible,  organized  tribe 

b.  Fie  believed  his  father  would  one  day  kill  him 

c.  Fie  struggled  with  adapting  to  new  social  situations 

d.  None  of  the  above 

18.  Hiring  discrimination  toward  military  spouses  is  legally  sanctioned  in  several  states. 

a.  True 

b.  False 

19.  In  the  military,  what  entity  is  responsible  for  responding  to  episodes  of  military  family  violence? 

a.  Family  Support  Centers 

b.  Family  Readiness  Groups 

c.  Family  Advocacy  Program 

d.  None  of  the  above,  it  is  referred  to  local  law  enforcement  officials 

20.  When  a  servicemember  dies,  how  should  a  parent  respond  to  a  young  child? 

a.  Keep  silent  about  the  death  until  the  child  is  older  and  able  to  deal  with  the  issue 

b.  Share  the  information  as  soon  as  possible  in  an  age-appropriate  manner 

c.  Request  that  a  social  worker  explain  the  death  to  the  child 

d.  Share  the  information  with  all  surviving  family  members  as  a  group 

Flow  extensive  would  you  say  is  your  overall  knowledge  of  military  culture  and  the  life  of  a 
servicemember  or  military  family? 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

1 

Very 

Little 

2 

3 

4 

5 

6 

7 

Very 

Extensive 

Appendix  F 

Health  Challenges  for  Wounded  Warriors  and  Their  Caregivers  Knowledge  Pretest 
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Health  Challenges  for  Wounded  Warriors  and  their  Caregivers 
Knowledge  Pretest 


Please  select  the  best  answer  for  each  of  the  following  questions. 

1.  Which  of  the  following  is  a  characteristic  of  complicated  grief,  but  not  major  depression  or 
PTSD? 

a)  Ruminations  about  past  failures 

b)  Fear 

c)  Diffuse  sadness 

d)  Separation  distress 

2.  Research  indicates  that  the  involvement  of  partners  and  close  family  members  in 

treatment  for  PTSD  is  beneficial. 

a)  True 

b)  False 

3.  The  acronym  VA  stands  for: 

a)  Veterans  Health  Administration 

b)  Veterans  Administration 

c)  Department  of  Veterans  Affairs 

d)  None  of  the  above 

4.  Rapport  is  best  achieved  by  avoiding  direct  questions  about  an  obvious  wound/injury  until  the 
person  initiates  the  conversation. 

a)  True 

b)  False 

5.  Who  is  not  eligible  for  most  VA  benefits? 

a)  Veteran  with  a  dishonorable  discharge 

b)  Surviving  spouse,  child  or  parent  of  a  deceased  Veteran 

c)  Uniformed  service  member 

d)  Present  or  former  reservist  or  National  Guard  member 

6.  What  is  DD  Form  214? 

a)  Certificate  of  Release  or  Discharge  from  Active  Duty 

b)  Verification  of  Military  Experience  and  Training 

c)  Pre-separation  Counseling  Checklist 

d)  The  DoD's  Disability  Determination  form 
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7.  Tertiary  Blast  Injuries: 

a)  Are  injuries  from  fragments  and  other  missiles 

b)  Result  from  displacement  of  the  whole  body  by  combined  pressure  shock  wave  and  dynamic 
overpressure 

c)  Are  injuries  that  most  commonly  result  in  damage  to  parts  of  the  eye 

d)  Are  rare  in  present-day  war  environments 

8.  Which  of  the  following  is  a  community-based  program  of  the  VA  that  provides  a  broad  range  of 
counseling,  outreach,  and  referral  services  throughout  the  country  to  help  veterans  reintegrate 
into  civilian  life  after  combat? 

a)  Soldiers'  Project 

b)  Give-An-Hour 

c)  Vet  Centers 

d)  U.S.  VETS 

9.  Assessments  of  the  impact  of  a  disability  should  consider  the  stage  of  identity  of  the  person  in 
question. 

a)  True 

b)  False 

10.  The  Glasgow  Coma  Scale: 

a)  Measures  blood  loss  in  coma  patients  after  blast  injury. 

b)  Uses  a  bubble  chamber  to  assess  patient  discomfort  while  unconscious. 

c)  Aims  to  give  a  reliable,  objective  way  of  recording  the  conscious  state  of  a  person  for  initial 
as  well  as  subsequent  assessment. 

d)  Is  not  used  to  diagnose  symptoms  of  combat-related  TBI  because  it  has  not  yet  been  vetted 
by  the  Department  of  Defense  and  the  VA. 

11.  Which  of  the  following  strengths  may  inhibit  servicemembers  from  quickly  seeking  assistance 
from  injury-related  concerns? 

a)  Self-reliance 

b)  Deference  to  authority 

c)  Following  a  well-developed  protocol  for  a  variety  of  situations 

d)  Flyperviligance  in  dangerous  situations 

12.  Which  of  the  following  is  not  a  goal  of  the  DoD  and  VA's  Integrated  Disability  Evaluation  System 

(IDES)? 

a)  Provide  a  single  rating  for  DoD  and  VA  benefits 

b)  Reduce  the  severity  of  disability  ratings  system-wide 

c)  Prevent  gaps  between  active  duty  and  veteran  benefits 

d)  Ensure  disability  claims  are  already  filed  with  the  VA  when  the  servicemember 
transitions  from  active  duty  to  veteran  status 
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13.  Injury-related  chronic  pain  is  a  significant  factor  to  consider  when  assessing  suicide  risk. 

a)  True 

b)  False 

14.  How  many  Polytrauma  Rehabilitation  Centers  are  there  in  the  VA  Polytrauma  System  of  Care? 

a)  8 

b)  24 

c)  4 

d)  None,  patients  are  referred  to  private  Polytrauma  centers  outside  the  VA 

15.  An  Anaplastologist: 

a)  Deals  with  the  prosthetic  rehabilitation  of  an  absent,  disfigured,  or  malformed 
anatomically  critical  location  of  the  face  or  body. 

b)  Works  with  patients  who  have  suffered  a  traumatic  brain  injury  to  help  them  rebuild 
their  cognitive  functioning. 

c)  Is  a  physical  therapist  who  focuses  specifically  on  helping  patient  strengthen  the 
muscles  around  the  lumbar  spine. 

d)  Is  generally  not  covered  by  TRICARE. 

16.  Which  area  of  the  brain  is  involved  in  planning  complex  cognitive  behaviors  and  in  the 
expression  of  personality  and  appropriate  social  behavior? 

a)  Parietal  lobe 

b)  Corpus  callosum 

c)  Occipital  lobe 

d)  Prefrontal  cortex 

17.  Servicemembers  with  newly  acquired  disabilities  are  best  served  by  providers  who  have  some 
type  of  disability. 

a)  True 

b)  False 

18.  Which  of  the  following  are  forms  of  power  and  control  used  by  batterers? 

a)  Controlling  access  to  military  I.D.  card  of  spouses  and  children 

b)  Threatening,  playing  with,  or  cleaning  weapons  around  intimate  partners  and  children 

c)  Not  sharing  military  pay  or  financial  records 

d)  Blaming  intimate  partner  violence  and  abuse  on  job  stress  or  alcohol 

e)  All  of  the  above 
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19.  Brain-injured  servicemembers  should  rest  for  at  least  three  months  before  beginning  therapies. 

a)  True 

b)  False 


20.  What  percentage  of  hospitalizations/ER  visits  for  Traumatic  Brain  Injury  does  MILD  Traumatic 


Brain  Injury  account  for? 

a)  10% 

b)  80% 

c)  40% 

d)  None  -  Traumatic  Brain  Injury  is  not  a  diagnosis  recognized  by  most  civilian  hospitals 


How  extensive  would  you  say  is  your  overall  knowledge  of  military  culture  and  the  life  of  a 
servicemember  or  military  family? 
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Evaluation  Findings  from  the  Military  Culture  Continuing  Education  Course 

Fifteen  course  participants  were  surveyed  through  a  multidimensional  questionnaire  that  addressed 
satisfaction,  perceptions  of  course  content  and  delivery,  and  implications  for  professional  self  efficacy 
and  clinical  practice.  Twenty-four  open-  and  closed-ended  questions  were  accompanied  by  items 
seeking  to  obtain  a  brief  demographic  profile  of  respondents.  Following  is  a  summary  of  the  main 
findings. 

Participant  Characteristics 

Table  1  displays  demographic  breakdowns  of  the  individuals  completing  the  questionnaire1.  Participants 
were  relatively  equally  distributed  with  regard  to  age,  though  predominantly  female  (91%).  Nearly 
three-quarters  (73%)  identified  as  White.  Marriage  and  Family  Therapy  and  Social  Work  were  about 
evenly  represented  (55%  &  45%,  respectively),  with  no  individuals  representing  other  professional 
disciplines. 

With  regard  to  professional  qualifications,  almost  half  (46%)  are  currently  licensed,  with  another  36% 
registered  as  Social  Work  Associates  or  Marriage  and  Family  Therapy  Interns.  About  half  (46%)  are 
relatively  new  clinicians,  reporting  less  than  five  years  of  postgraduate-level  clinical  practice,  though 
27%  reported  having  at  least  15  years  experience.  Approximately  two-thirds  (64%)  work  full-time  in 
behavioral  health,  representing  the  fields  of  aging,  child  welfare,  disabilities,  health,  mental  health,  and 
substance  abuse.  Though  these  individuals  are  generally  focused  on  direct  practice  (64%  indicated  all  or 
nearly  all  of  their  time  is  spent  here),  46%  indicated  that  little  (i.e.,  less  than  10%)  of  their  time  is  spent 
working  with  military-related  populations. 


Table  1.  Characteristics  of  course  evaluation  participants  (n=ll) 


Characteristic 

N 

Percentage 

Age 

25-34 

2 

18% 

35-44 

3 

27% 

45-54 

3 

27% 

55-64 

3 

27% 

Gender 

Female 

10 

91% 

Male 

1 

9% 

Race  and  Ethnicity 

Black/African-American 

1 

9% 

Hispanic/Latino(a) 

2 

18% 

1  It  should  be  noted  that  n=ll  respondents  for  the  demographic  section,  as  four  individuals  did  not  complete  the 
demographic  items.  This  might  have  been  an  inadvertent  oversight,  as  the  items  were  located  on  the  reverse  side 
of  the  questionnaire's  final  page. 
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White 

8 

73% 

Professional  Discipline 

Marriage  &  Family  Therapist 

6 

55% 

Social  Worker 

5 

46% 

Licensure  Status 

Licensed 

5 

46% 

In  process  (registered) 

4 

36% 

Not  licensed 

2 

18% 

Years  of  Postgraduate  Practice 

Less  than  5 

5 

46% 

5-9 

1 

9% 

10-14 

2 

18% 

15-19 

1 

9% 

20  or  more 

2 

18% 

Employment  Status 

Full-time  behavioral  health 

7 

64% 

Employed,  non-  behavioral  health 

3 

27% 

Not  currently  employed 

1 

9% 

Practice  Setting(s)* 

Academia 

1 

10% 

Aging 

1 

10% 

Child  Welfare 

1 

10% 

Disabilities 

1 

10% 

Health 

1 

10% 

Mental  Health 

7 

70% 

Substance  Abuse 

2 

20% 

Percentage  of  Time  Spent  in  Direct  Practice 

All  or  nearly  all  (>90%) 

7 

64% 

Most  (51-90%) 

0 

0% 

Some  (11-50%) 

2 

18% 

Little  (<10%) 

2 

18% 

Percentage  of  Time  with  Military  Populations 

All  or  nearly  all  (>90%) 

4 

36% 

Most  (51-90%) 

0 

0% 

Some  (11-50%) 

2 

18% 

Little  (<10%) 

5 

46% 

*  Adds  to  greater  than  100%,  as  selection  of  multiple  categories  was  permitted 
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Evaluation  Model 

Literature  within  the  behavioral  health  sciences  notes  challenges  in  effectively  measuring  clinician 
training  outcomes.  Training  issues  surround  the  appropriate  method  of  training  (Herschell,  et  al 2010) 
and  transfer  of  skill  to  actual  practice  (Clarke,  et  al.,  2002).  Academic  researchers  across  an  array  of 
domains  from  psychology,  human  services,  and  higher  education  programming  often  cite  the  4-level 
Kirkpatrick  Training  Evaluation  Model  in  examining  the  effects  of  training  and  continuing  education  (e.g., 
Praslova,  2010,  Clarke  2002).  The  Kirkpatrick  model's  dominance  in  the  training  field  leaves  it  ideally 
suited  as  a  framework  in  which  to  consider  CIR's  CE  training  series  as  it  will  be  familiar  to  a  variety  of 
potential  partners  within  the  higher  education,  government,  and  business  communities. 

The  levels  of  the  Kirkpatrick  Model  are  described  below  in  Table  2.  Briefly,  the  model  posits  4  levels  of 
evaluation:  reaction  criteria,  learning  criteria,  behavior  criteria,  and  results  criteria  (Kirkpatrick,  1996; 
Praslova,  2010).  Reaction  and  learning  criteria  are  considered  internal  criteria,  as  they  focus  on  changes 
that  occur  within  the  training  program  itself.  Behavioral  and  results  criteria  are  considered  external 
criteria  and  occur  after  the  training;  behavioral  and  results  criteria  are  influenced  by  factors  beyond  the 
scope  of  the  training  (e.g.,  organizational  context,  opportunity/support  to  use  and  develop  skills). 


Table  2.  Kirkpatrick's  4  Levels  of  Training  Evaluation. 


Level 

Criteria 

Locus  of  Change 

Variable  measured 

Timepoint 

measured 

1 

Reaction 

Internal  to  training 

Affective  (satisfaction) 

&  Utility  (plans  to 
implement)  judgments 

At  training 

2 

Learning 

Internal  to  training 

Knowledge  &  skill 
gains 

At  training 

3 

Behavioral 

External  to  training 

On-the-job 

performance 

Follow-up 

4 

Results 

External  to  training 

Productivity  gains, 
organizational  change 

Follow-up 

3 
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Level  1:  Reaction  Criteria 

Reaction  criteria  consist  of  two  elements  internal  to  the  training  itself:  affective  reaction  and  utility 
judgments.  Affective  reaction  refers  to  participants'  satisfaction  and  enjoyment  of  the  course.  Utility 
judgments  refer  to  participants'  beliefs  about  how  much  they  learned  and  how  they  plan  to  use  what 
they  learn  in  their  practices  with  veterans  and  military  families. 

Reaction  criteria  were  assessed  via  eighteen  Likert-type  items  regarding  perceptions  of  course  content 
and  delivery  and  participants'  intentions  to  use  knowledge  learned  during  the  course  in  clinical  practice. 
Each  item  was  rated  on  a  scale  of  1  (Strongly  agree)  to  5  (Strongly  disagree).  Two  items  were  reversed 
scored,  as  indicated  below. 

Items  were  analyzed  in  the  context  of  two  scales  -  one  assessing  participants'  affective  reaction  to  the 
course,  that  is  how  the  course  met  with  their  expectations  in  terms  of  quality  of  instruction,  breadth  and 
depth  of  material,  and  general  satisfaction,  and  the  other  assessing  participants'  utility  judgments  about 
the  course  -  that  is,  participants'  beliefs  that  they  will  use  what  they  learned  in  clinical  practice  with 
military-related  populations  and  that  they  gained  agency  in  working  effectively  with  military 
populations. 

Affective  Reaction:  Perceptions  of  Course  Content  and  Delivery 

Thirteen  items  assessing  satisfaction  with  the  course  content,  delivery,  and  overall  value  of  the  course 
were  combined  (via  averaging)  into  a  scale,  which  demonstrated  reliability  above  the  acceptable  level 
(Cronbach's  a=  .83).  Overall,  the  average  response  to  the  affective  reaction  scale  was  4.25,  out  of  a 
possible  5,  (range:  3.62  to  4.77),  indicating  a  high  level  of  satisfaction  with  the  course  content,  delivery, 
and  overall  value.  Mean  response  by  item,  as  well  as  response  frequency  by  item  are  presented  in  Table 
3. 


Table  3.  Perceptions  of  course  content  and  delivery  (n=15  unless  otherwise  noted) 


Item  (Mean  Response) 

N 

Percent 

The  instructors  were  interesting  &  kept  my 
attention  throughout  the  training.  (4.46) 

Strongly  agree 

7 

46.7 

Agree 

8 

53.3 

Neither  agree  nor  disagree 

- 

- 

Disagree 

- 

- 

Strongly  disagree 

- 

- 

The  material  was  clearly  presented.  (4.23) 

Strongly  agree 

6 

40.0 

Agree 

8 

53.3 

Neither  agree  nor  disagree 

- 

- 

Disagree 

1 

6.7 
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Strongly  disagree 

- 

- 

The  balance  of  instruction  methods  (lecture, 
discussion,  small  group  work)  was  about  right. 
(3.85) 

Strongly  agree 

2 

13.3 

Agree 

9 

60.0 

Neither  agree  nor  disagree 

2 

13.3 

Disagree 

2 

13.3 

Strongly  disagree 

- 

The  activities  and  exercises  aided  in  my  learning. 
(4.54) 

Strongly  agree 

7 

46.7 

Agree 

8 

53.3 

Neither  agree  nor  disagree 

- 

Disagree 

- 

Strongly  disagree 

- 

The  breadth  of  material  covered  was  appropriate. 
(4.15) 

Strongly  agree 

6 

40.0 

Agree 

7 

46.7 

Neither  agree  nor  disagree 

1 

6.7 

Disagree 

1 

6.7 

Strongly  disagree 

- 

The  depth  of  the  material  covered  was 
appropriate.  (3.69) 

Strongly  agree 

4 

26.7 

Agree 

7 

46.7 

Neither  agree  nor  disagree 

1 

6.7 

Disagree 

3 

20.0 

Strongly  disagree 

- 

This  course  was  delivered  at  an  appropriate  level 
for  practitioners  with  my  level  of  experience. 

(4.31) 

Strongly  agree 

6 

40.0 

Agree 

7 

46.7 

Neither  agree  nor  disagree 

2 

13.3 

Disagree 

- 

Strongly  disagree 

- 

It  felt  like  there  was  a  disconnect  between  the 
instructors'  perspectives  (having  served)  and 
mine,  as  a  civilian  provider  (R).*  (3.85) 

Strongly  agree 

3 

20.0 

Agree 

8 

53.3 

Neither  agree  nor  disagree 

1 

6.7 

Disagree 

2 

13.3 

Strongly  disagree 

1 

6.7 
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The  course  covered  the  right  amount  of 
information  for  the  length  of  the  training.  (3.62) 

(n  =  13) 

Strongly  agree 

3 

23.1 

Agree 

Neither  agree  nor  disagree 

6 

46.2 

Disagree 

4 

30.8 

Strongly  disagree 

- 

The  topics  covered  were  what  1  was  hoping  to  see 
in  this  course.  (4.31) 

(n  =  13) 

Strongly  agree 

5 

38.5 

Agree 

7 

53.8 

Neither  agree  nor  disagree 

1 

7.7 

Disagree 

- 

Strongly  Disagree 

- 

Overall,  the  course  was  valuable.  (4.77) 

(n  =  13) 

Strongly  agree 

10 

76.9 

Agree 

Neither  agree  nor  disagree 

3 

23.1 

Disagree 

- 

Strongly  Disagree 

- 

A  course  on  military  culture  should  be 
compulsory  for  practitioners  working  with 
veterans  and  military  families.  (4.69) 

(n  =  13) 

Strongly  agree 

10 

76.9 

Agree 

2 

15.4 

Neither  agree  nor  disagree 

1 

7.7 

Disagree 

- 

Strongly  Disagree 

- 

1  would  recommend  this  course  to  a  friend  or 
colleague  interested  in  working  with  veterans  and 
military  families.  (4.77) 

(n  =  13) 

Strongly  agree 

10 

76.9 

Agree 

Neither  agree  nor  disagree 

3 

23.1 

Disagree 

- 

Strongly  Disagree 

- 

As  shown  above,  items  with  the  most  favorable  responses  regarded  the  overall  value  of  the  course  and 
the  need  for  a  course  like  this  to  be  compulsory  for  practitioners  working  with  veterans  and  military 
families.  This  sentiment  is  consistent  with  results  from  a  survey  from  several  hundred  behavioral  health 
providers  working  with  veterans  and  military  families  (Nissly,  2011  -  personal  communication).  In 
addition,  when  asked  to  supply  the  single  most  important  thing  learned  in  training,  several  responses 
indicated  a  distinct  appreciation  for  the  military  as  a  culture  crucial  to  consider  as  a  treating  clinician.  For 
example,  some  respondents  indicated: 
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treat  individuals  individually;  not  everyone's  experience  is  the  same;  culture  is  somewhat  individually 
determined 

my  own  biases  about  the  military  &  how  they  can  impact  my  work 

take  each  case  individually;  impact  on  the  families;  military  as  its  own  culture 

Items  with  less  favorable,  but  nonetheless  positive,  responses  addressed  the  depth  of  information 
presented,  the  amount  of  information  presented,  the  balance  of  methods  used  in  the  course,  and 
participants'  ability  to  relate  to  veteran  and  active  duty  course  instructors.  Responses  to  open  ended 
items  shed  more  light  on  this  feedback  -  several  participants  felt  additional  time  was  needed  to  cover 
the  amount  of  material  in  the  appropriate  depth  -  especially  regarding  professional  ethics  around 
treating  servicemembers,  veterans,  and  their  families.  Participants  also  expressed  an  interest  in  learning 
more  about  women  in  the  military  and  gay  and  lesbian  servicemembers.  Further,  participants  indicated 
a  desire  for  more  time  spent  on  military  rank  and  promotion.  These  responses  were  consistent  with 
responses  from  the  focus  group  held  immediately  following  the  course.  For  example  one  respondent  in 
the  focus  group  indicated: 

I  didn't  have  any  military  knowledge  -  lost  track  during  Vat's  presentation.  Need  more  info  on  how 
military  works  and  how  ranks  work  -  instead  of  jumping  into  how  the  training  works.  I'm  still  confused 
by  some  of  the  logistics  of  the  military. 

Regarding  balance  of  instruction  methods,  participants  expressed  a  desire  for  more  time  spent  in  small 
group  work  and  interactive  learning  experiences. 

As  another  indicator  of  perceived  value  of  the  course,  participants  were  asked  to  indicate  the 
appropriate  cost  for  the  course.  Of  the  9  respondents  who  completed  this  question,  most  responded 
that  the  appropriate  fee  for  the  10  hour  course  would  be  between  $75  -  150.  See  Chart  1: 

Chart  1.  Responses  to  "I  would  consider  $ _ to  be  a  suitable  cost  for  this  course  as  a  Continuing 

Education  course." 
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Utility  Judgments:  Implications  for  Clinical  Practice 

Six  items  assessing  participants'  beliefs  and  intentions  regarding  applying  course  concepts  to  practice 
were  examined  for  internal  reliability.  Results  indicated  that  the  first  five  items  were  appropriate  to 
combine  (via  averaging)  into  a  scale,  which  demonstrated  reliability  above  the  acceptable  level 
(Cronbach's  a=  .83).  The  item,  "After  this  course,  I  am  confident  about  where  to  look  for  additional 
resources  and  information  on  working  with  veteran  and  military  family  clients,"  was  not  included  in  the 
utility  scale.  Overall,  the  average  response  to  the  utility  judgments  scale  was  4.26,  out  of  a  possible  5, 
(range:  3.79  to  4.43),  indicating  a  strong  sense  of  intention  to  apply  course  concepts  to  practice  with 
military  clients.  Mean  response  by  item,  as  well  as  response  frequency  by  item  are  presented  in  Table  4. 


Table  4.  Perceived  influence  of  Military  Cultures  course  on  clinical  practice  (n  =  15  unless  otherwise 
noted). 


Item  (Mean  response) 

N 

Percent 

This  course  made  me  aware  of  the  differences  in 
civilian  and  military  culture  with  regard  to 
behavioral  health  practice  issues.  (4.36) 

Strongly  agree 

6 

40 

Agree 

Neither  agree  nor  disagree 

Disagree 

Strongly  disagree 

9 

60 

As  a  result  of  what  1  learned  in  this  course,  1  can 
think  of  two  concrete  strategies  1  can  use  in  my 
practice  with  veteran  clients.  (3.79) 

Strongly  agree 

4 

26.7 

Agree 

8 

53.3 

Neither  agree  nor  disagree 

1 

6.7 

Disagree 

1 

6.7 

Strongly  disagree 

1 

6.7 

Participation  in  this  course  will  impact  how  1 
interact  with  my  military-related  clients. (4. 43) 

Strongly  agree 

8 

53.3 

Agree 

6 

40 

Neither  agree  nor  disagree 

1 

6.7 

Disagree 

- 

Strongly  disagree 

- 

As  a  result  of  this  training,  1  will  be  more 
confident  in  seeking  out  opportunities  to  work 
with  veterans  and  military  families.  (4.29) 

Strongly  agree 

7 

46.7 

Agree 

7 

46.7 
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Neither  agree  nor  disagree 

- 

Disagree 

1 

6.7 

Strongly  disagree 

- 

Today's  training  made  me  feel  less  equipped  to 

(n  =  14) 

practice  effectively  with  veterans  and  military 

families  (R).*  (4.43) 

Strongly  agree 

7 

50 

Agree 

6 

42.9 

Neither  agree  nor  disagree 

1 

7.1 

Disagree 

Strongly  disagree 

After  this  course,  1  am  confident  about  where  to 

look  for  additional  resources  and  information  on 

working  with  veteran  and  military  family  clients. 

(4.20) 

Strongly  agree 

3 

20 

Agree 

12 

80 

Neither  agree  nor  disagree 

- 

Disagree 

- 

Strongly  disagree 

' 

Level  2: Learning 

Level  2,  Learning  criteria,  of  the  Kirkpatrick  Model  refers  to  the  knowledge  acquired,  skills  improved,  or 
attitudes  changed  due  to  training  (Kirkpatrick,  1996).  Learning  criteria  are  internal  to  the  training  and 
are  typically  measured  immediately  after  the  training  occurs.  Because  the  Military  Cultures  CEU  course 
sought  to  introduce  participants  to  the  military  lifestyle  as  a  culture  in  which  a  successful  practitioner 
will  gain  competence  in  order  to  practice  effectively,  a  pre/post  test  of  military-related  knowledge  was 
used  as  a  measure  of  learning. 

Pre/Post  Knowledge  Gains 

A  20-item  multiple  choice  &  true/false  measure  assessed  participant  gains  in  knowledge  of  military 
culture  as  a  result  of  the  10-hour  course.  Results  indicated  that  overall,  participants  scored  higher  on  the 
knowledge  measure  at  the  conclusion  of  the  course  (M  =  13.00,  SD  =  2.16)  than  at  the  beginning  of  the 
course  (M  =  10.23,  SD  =  2.49),  t(12)  =  4.62,  p  <  .05. 

Because  the  items  were  developed  independently  of  the  course  presentations,  two  independent  raters 
marked  which  items  were  covered  in  the  course  presentations.  A  third  rater  acted  as  a  tie-breaker, 
ultimately  arriving  at  a  list  of  9  "valid"  items.  Comparing  participants'  pre/post  scores  on  the  valid 
items,  results  again  indicated  that  participants  scored  higher  on  the  knowledge  measure  at  the  end  of 
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the  course  (M  =  6.15,  SD  =  1.21)  than  at  the  beginning  of  the  course  (A/7  =  4.46,  SD  =  1.21),  t(  12)  =  3.94,  p 
<.05. 

Between  Group  Comparisons 

In  accordance  with  the  Non  Human  Subjects  Research  (NHSR)  classification  of  evaluating  the  CEU 
courses,  we  are  permitted  to  analyze  the  data  for  feedback  regarding  course  content,  rather  than  for 
purposes  of  making  generalizations  about  course  participants.  Nonetheless,  between  group 
comparisons  were  cursorily  explored  using  a  series  of  independent  sample  t-tests.  Dependent  variables 
of  interest  included:  responses  to  affective  measures,  responses  to  utility  measures,  and  pre/posttest 
knowledge  gains.  No  significant  differences  emerged  for  these  outcomes  based  on:  professional 
discipline  (i.e.,  MFT,  Social  Worker),  years  practicing  (i.e.,  more  than  10,  less  than  10),  or  amount  of 
practice  spent  with  military-related  clients  (i.e.,  most  practice  with  military-clients,  very  little  practice 
with  military  clients).  See  Table  5  for  means  and  standard  deviations. 


Table  5.  Group  means  and  standard  deviations  for  Affect,  Utility,  and  Knowledge  Measures. 


N 

Affect  Mean  (SD) 

Utility  Mean  (SD) 

Pre/post 

knowledge  Mean 
(SD) 

Professional 

Discipline 

Social  Worker 

5 

4.15  (.40) 

4.30  (.35) 

4.25  (1.5) 

MFT 

6 

4.29  (.51) 

4.07  (.83) 

1.80  (2.59) 

Years  Practicing 

<  10 

6 

4.12  (.41) 

4.00  (.81) 

3.00(3.56) 

>10 

5 

4.36  (.47) 

4.45  (.32) 

2.67  (1.50) 

Practice  with 

Military  Clients 

Most 

4 

4.00  (.27) 

4.10  (.26) 

3.50  (2.12) 

Very  little 

7 

4.36  (.49) 

4.20  (.86) 

2.71  (2.63) 
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Summary 

In  conclusion,  results  from  the  first,  on-ground  pre-pilot  CEU  course  on  Military  Culture  are  promising. 
Available  data  on  Levels  1  and  2  of  the  Kirkpatrick  Training  Evaluation  Model  suggest  that  participants 
were  highly  satisfied  with  the  course,  participants  believe  knowledge  gained  in  the  course  will  affect 
how  they  approach  clinical  practice  with  veterans  and  military  families,  and  participants  demonstrated 
gains  in  knowledge  regarding  military  culture  over  the  10  hours  of  instruction.  No  statistically  significant 
differences  emerged  on  these  three  main  domains  by  profession,  experience,  or  amount  of  practice 
dedicated  to  military  clients.  With  regard  to  refining  the  course,  participants  requested  that  knowledge 
be  presented  at  a  more  basic  level  and  that  more  interactive  training  modalities  be  used  in  instruction. 
In  addition,  numerous  participants  requested  additional  information  on  ethical  considerations  in 
practice  when  treating  veterans  or  military  families. 

Next  Steps 


Level  3:  Behavior 

The  third  level  of  the  Kirkpatrick  Evaluation  Model  measures  the  extent  participants  change  their  on- 
the-job  behavior  as  a  result  of  the  training  (Kirkpatrick,  1996).  The  use  of  training  knowledge  and  skills  is 
commonly  referred  to  as  transfer  of  training.  Transfer  is  considered  an  external  criteria  and  is  frequently 
influenced  by  opportunities  within  one's  work  setting  to  practice  new  skills  (Clarke,  2002). 

As  transfer  is  conceptualized  to  occur  after  the  training,  it  is  best  measured  at  follow-up. 

To  assess  the  extent  to  which  participants  in  the  Military  Cultures  CEU  course  transfer  new  knowledge 
and  skills  to  their  clinical  practices,  it  is  recommended  that  participants  be  queried  via  internet  survey  at 
3  and  6  months  post  course  participation  (i.e.,  early  June  2011,  early  September  2011  respectively). 
Following  Clarke  (2002),  domains  on  which  transfer  may  be  assessed  are: 

•  participants'  beliefs  about  the  benefits  gained  as  a  result  of  the  course 

•  evidence  of  using  the  training  in  practice  with  military-related  clients 

•  barriers  to  implementing  knowledge  and  skills  learned  in  the  training 

•  factors  that  facilitated  transfer  to  practice  with  military-clients 

At  follow-up,  participants  could  also  be  asked  to  review  their  records  for  the  last  year  to  provide  typical 
attrition  rates  for  military-related  clients  for  the  6  months  prior  to,  and  the  6  months  following,  their 
participation  in  the  Military  Cultures  CEU  course. 


Level  4:  Results 

Results  criteria  refer  to  the  final  results  that  occur  due  to  training  (Kirkpatrick,  1996)  and  extend  beyond 
the  individual  participant  in  training  to  broader  organizational  change  (e.g.,  productivity  gains,  increased 
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employee  morale)  (Praslova,  2010).  Results  criteria  are  used  less  frequently  than  the  other  levels  of  the 

Kirkpatrick  model  as  they  are  difficult  to  evaluate  (Praslova,  2010). 

In  the  current  evaluation,  results  criteria  can  be  measured  in  a  number  of  ways.  At  the  3  and  6  month 

follow-up,  participants  can  be  queried  regarding: 

•  the  extent  to  which  they  have  changed  the  way  they  train  other  clinicians  in  their  practice  who 
work  with  military-related  clients 

•  changes  in  training  requirements  for  clinicians  in  their  organization  who  work  with  military- 
related  clients 

•  attrition  rates  for  military-related  clients  for  the  6  months  prior  to,  and  the  6  months  following, 
their  participation  in  the  Military  Cultures  CEU  course 

•  client-to-client  referral  rates  for  military-related  clients  have  changed  for  the  6  months  prior  to, 
and  the  6  months  following,  their  participation  in  the  Military  Cultures  CEU  course 

Results  criteria  can  be  measured  in  more  depth  in  the  Randomized  Control  Trial  (RCT)  planned  for  2012. 

Recommendations 

•  The  RCT  pilot  scheduled  for  2012  may  benefit  by  incorporating  suggestions  from  the  Advisory 
Board  meetings  held  in  March  2011,  as  well  as  findings  from  clinical  training  literature  indicating 
that  a  coaching  competent  included  in  training  improves  adoption  of  innovation,  skill  retention, 
and  client  outcomes,  compared  to  workshops  alone  (Herschell,  et  al.,  2010). 

•  Future  CEU  courses  in  the  series  may  benefit  by  incorporating  more  pictures  and  video  footage 
from  combat  zones.  This  feedback  was  incorporated  into  the  second  CEU  course,  Clinical 
Practice  with  Servicemembers  and  Veterans. 

•  Future  introductory  courses  on  military  culture  should  spend  additional  time  on  ethical 
considerations  for  treatment,  as  well  as  ensure  veteran  and/or  active  duty  instructors  be 
cautious  of  the  use  of  acronyms  or  military  jargon  in  their  presentations. 
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Evaluation  Findings:  Clinical  Practice  Continuing  Education  Course 

Eighteen  behavioral  health  providers  participated  in  the  second  continuing  education  course  offered  by 
CIR;  Clinical  Practice  with  Servicemembers  and  Veterans.  Participants  were  surveyed  through  a 
multidimensional  questionnaire  that  addressed  satisfaction,  perceptions  of  course  content  and  delivery, 
and  implications  for  professional  self  efficacy  and  clinical  practice.  Twenty-four  open-  and  closed-ended 
questions,  as  well  as  items  measuring  the  demographic  profile  of  respondents,  comprised  the  evaluation 
measure.  Following  is  a  summary  of  the  main  findings. 

Participant  Characteristics 

Table  1  displays  demographic  breakdowns  of  the  individuals  in  attendance1.  Participants  were  relatively 
equally  distributed  with  regard  to  age  and  gender  (61%  female).  Nearly  three-quarters  (72%)  self- 
identified  as  White.  Marriage  and  Family  Therapy  and  Social  Work  were  about  evenly  represented  (44% 
&  39%,  respectively),  with  individuals  identifying  themselves  as  psychologists,  psychiatrists,  and  nurse 
practitioners  as  well. 

With  regard  to  professional  qualifications,  two-thirds  (67%)  are  currently  licensed,  with  nearly  another 
third  (28%)  registered  as  Social  Work  Associates  or  Marriage  and  Family  Therapy  Interns.  About  one- 
third  (33%)  are  relatively  new  clinicians,  reporting  less  than  five  years  of  postgraduate-level  clinical 
practice,  with  half  (50%)  reporting  over  20  years  experience.  Most  participants  (83%)  work  full-time  in 
behavioral  health,  representing  the  fields  of  academia,  aging,  child  welfare,  health,  mental  health,  and 
substance  abuse.  Though  these  individuals  are  generally  focused  on  direct  practice  (83%  indicated  all  or 
nearly  all  of  their  time  is  spent  here),  44%  indicated  that  little  (i.e.,  less  than  10%)  of  their  time  is  spent 
working  with  military-related  populations. 


Table  1.  Characteristics  of  course  evaluation  participants  (n=18) 


Characteristic 

N 

Percentage 

Age 

25-34 

5 

27.8 

35-44 

1 

5.6 

45-54 

5 

27.8 

55-64 

5 

27.8 

65+ 

2 

11.1 

Gender 

Female 

11 

61.1 

Male 

7 

38.9 

Race  and  Ethnicity 

Asian/Pacific  Islander 

1 

5.6 

1  N  =18  respondents  for  the  demographic  section  reflects  the  18  participants  who  registered  and  attended  the 
course  on  Day  1.  However,  n  =  16  for  the  survey  questionnaires  as  two  individuals  present  on  Day  1  were  not  able 
to  remain  for  Day  2  due  to  personal  conflicts. 
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Black/African-American 

Hispanic/Latino(a) 

Anglo/White 

Other 

1 

2 

13 

1 

5.6 

11.1 

72.2 

5.6 

Professional  Discipline 

Marriage  &  Family  Therapist 

8 

44.4 

Psychologist 

1 

5.6 

Social  Worker 

7 

38.9 

Other  (Psychiatrist,  Nurse  Practitioner) 

2 

11.1 

Licensure  Status 

Licensed 

12 

66.7 

In  process  (registered) 

5 

27.8 

Not  licensed 

1 

5.6 

Years  of  Postgraduate  Practice 

Less  than  5 

6 

33.3 

5-9 

1 

5.6 

10-14 

1 

5.6 

15-19 

1 

5.6 

20  or  more 

9 

50.0 

Employment  Status 

Full-time  behavioral  health 

15 

83.3 

Part-time  behavioral  health 

2 

11.1 

Employed,  non-  behavioral  health 

1 

5.6 

Practice  Setting(s)* 

Academia 

2 

- 

Aging 

1 

- 

Child  Welfare 

2 

- 

Disabilities 

- 

- 

Health 

3 

- 

Mental  Health 

11 

- 

Substance  Abuse 

4 

- 

Percentage  of  Time  Spent  in  Direct  Practice 

All  or  nearly  all  (>90%) 

8 

44.4 

Most  (51-90%) 

7 

38.9 

Some  (11-50%) 

1 

5.6 

Little  (<10%) 

2 

11.1 

Percentage  of  Time  with  Military  Populations 

All 

5 

27.8 

Some  (11-50%) 

5 

27.8 

Little  (<10%) 

8 

44.4 

*  Adds  to  more  than  18  cases,  as  selection  of  multiple  practice  settings  was  permitted 
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Evaluation  Model 

The  4-level  Kirkpatrick  Training  Evaluation  Model  is  used  as  a  framework  with  which  to  consider  CIR's  CE 
training  series.  The  levels  of  the  Kirkpatrick  Model  are  described  below  in  Table  2.  The  model  posits  4 
levels  of  evaluation:  reaction  criteria,  learning  criteria,  behavior  criteria,  and  results  criteria  (Kirkpatrick, 
1996;  Praslova,  2010).  Reaction  and  learning  criteria  are  considered  internal  criteria,  as  they  focus  on 
changes  that  occur  within  the  training  program  itself.  Behavioral  and  results  criteria  are  considered 
external  criteria  and  occur  after  the  training;  behavioral  and  results  criteria  are  influenced  by  factors 
beyond  the  scope  of  the  training  (e.g.,  organizational  context,  opportunity/support  to  use  and  develop 
skills).  For  a  more  complete  description  of  the  Kirkpatrick  model,  please  see  the  Evaluation  Report  1  on 
the  Military  Culture  Course. 

Table  2.  Kirkpatrick's  4  Levels  of  Training  Evaluation. 


Level 

Criteria 

Locus  of  Change 

Variable  measured 

Timepoint 

measured 

1 

Reaction 

Internal  to  training 

Affective  (satisfaction) 

&  Utility  (plans  to 
implement)  judgments 

At  training 

2 

Learning 

Internal  to  training 

Knowledge  &  skill 
gains 

At  training 

3 

Behavioral 

External  to  training 

On-the-job 

performance 

Follow-up 

4 

Results 

External  to  training 

Productivity  gains, 
organizational  change 

Follow-up 

Level  1:  Reaction  Criteria 

Reaction  criteria  consist  of  two  elements  internal  to  the  training  itself:  affective  reaction  and  utility 
judgments.  Affective  reaction  refers  to  participants'  satisfaction  and  enjoyment  of  the  course.  Utility 
judgments  refer  to  participants'  beliefs  about  how  much  they  learned  and  how  they  plan  to  use  what 
they  learn  in  their  practices  with  veterans  and  military  families. 
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Reaction  criteria  were  assessed  via  seventeen  Likert-type  items  regarding  perceptions  of  course  content 
and  delivery  and  participants'  intentions  to  use  knowledge  learned  during  the  course  in  clinical  practice. 
Each  item  was  rated  on  a  scale  of  1  ( Strongly  agree)  to  5  (Strongly  disagree). 

Items  were  analyzed  in  the  context  of  two  scales  -  one  assessing  participants'  affective  reaction  to  the 
course,  that  is  how  the  course  met  with  their  expectations  in  terms  of  quality  of  instruction,  breadth  and 
depth  of  material,  and  general  satisfaction,  and  the  other  assessing  participants'  utility  judgments  about 
the  course  -  that  is,  participants'  beliefs  that  they  will  use  what  they  learned  in  clinical  practice  with 
military-related  populations  and  that  they  gained  agency  in  working  effectively  with  military 
populations. 


Affective  Reaction:  Perceptions  of  Course  Content  and  Delivery 

Twelve  items  assessing  satisfaction  with  the  course  content,  delivery,  and  overall  value  of  the  course 
were  combined  (via  averaging)  into  a  scale,  which  demonstrated  reliability  at  the  acceptable  level 
(Cronbach's  a=  .69).  Overall,  the  average  response  to  the  affective  reaction  scale  was  4.52,  out  of  a 
possible  5,  (range:  3.91  to  4.91),  indicating  a  high  level  of  satisfaction  with  the  course  content,  delivery, 
and  overall  value.  Mean  response  by  item,  as  well  as  response  frequency  by  item  are  presented  in  Table 
3. 


Table  3.  Perceptions  of  course  content  and  delivery  (n=16  unless  otherwise  noted) 


Item  (Mean  Response) 

N 

Percent 

Ql.  The  instructors  were  interesting  &  kept  my 

attention  throughout  the  training.  (4.82) 

Agree 

4 

25.0 

Strongly  Agree 

12 

75.0 

Q2.  The  material  was  clearly  presented.  (4.73) 

Agree 

5 

31.3 

Strongly  Agree 

11 

68.8 

Q3.  The  balance  of  instruction  methods  (lecture, 

discussion,  small  group  work)  was  about  right. 

(3.91) 

Disagree 

1 

6.3 

Neither  agree  nor  disagree 

2 

12.5 

Agree 

8 

50.0 

Strongly  Agree 

5 

31.3 

Q4.  The  activities  and  exercises  aided  in  my 

learning.  (4.00) 

Neither  agree  nor  disagree 

5 

31.3 

Agree 

4 

25.0 

Strongly  Agree 

7 

43.8 
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Q5.  The  breadth  of  material  covered  was 
appropriate.  (4.63) 

Agree 

5 

33.3 

Strongly  agree 

10 

66.7 

Q6.  The  depth  of  the  material  covered  was 
appropriate.  (4.73) 

Agree 

6 

40.0 

Strongly  agree 

9 

60.0 

Q7.  This  course  was  delivered  at  an  appropriate 
level  for  practitioners  with  my  level  of 
experience.  (4.64) 

Neither  agree  nor  disagree 

1 

6.7 

Agree 

6 

40.0 

Strongly  agree 

8 

53.3 

Q17.  The  course  covered  the  right  amount  of 
information  for  the  length  of  the  training.  (4.36) 

Disagree 

1 

8.3 

Agree 

5 

41.7 

Strongly  disagree 

6 

50.0 

Q18.  The  topics  covered  were  what  1  was  hoping 
to  see  in  this  course.  (4.45) 

Neither  agree  nor  disagree 

1 

8.3 

Agree 

4 

33.3 

Strongly  agree 

7 

58.3 

Q21.  Overall,  the  course  was  valuable.  (4.55) 

Strongly  disagree 

1 

8.3 

Agree 

1 

8.3 

Strongly  agree 

10 

83.3 

Q22.  A  course  on  military  culture  should  be 
compulsory  for  practitioners  working  with 
veterans  and  military  families.  (4.55) 

Strongly  disagree 

1 

9.1 

Agree 

4 

9.1 

Strongly  agree 

5 

81.8 

Q23. 1  would  recommend  this  course  to  a  friend 
or  colleague  interested  in  working  with  veterans 
and  military  families.  (4.91) 

Agree 

1 

8.3 

Strongly  agree 

11 

91.7 

As  shown  above,  items  with  the  most  favorable  responses  regarded  the  overall  value  of  the  course  and 
the  ability  of  the  instructors  to  engage  with  class  participants.  Although  still  favorable,  items  with  the 
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least  strong  endorsements  inquired  about  the  balance  of  instruction  methods  and  the  extent  to  which 
course  activities  and  exercises  contributed  to  learning.  Additional  information  about  satisfaction  and 
expectation  was  obtained  through  participants'  responses  to  2  open-ended  questions.  Responses  to 
those  items  are  presented  below: 

Please  describe  any  topics  that  you  expected  to  be  covered,  but  weren't. 

•  /  missed  the  1st  training  and  so  I  was  looking  for  info  on  all  those  initials! 

•  More  about  stigma/homeless 

•  Course  descriptions  and  objectives  were  clear.  There  was  a  comment  regarding  "compassion 
fatigue"  which  seemed  appropriate  given  the  content  and  application. 

•  Any  additional  training  in  different  modalities  of  therapy? 

•  Student  vets,  vet  to  vet,  staff  and  non-military  education,  and  welcome  home. 

•  Some  more  direct  info  re:  more  hands-on  skills  for  tx  with  vets  related  to  prob-solving,  decision 
making  and  goal  setting  as  well  as  "seif"  dev. 

•  Bio-feedback  work  and  use  in  management  of  PTSD  symptoms.  MST  Military  served  [sic]  trauma 
issues  -  anything  (?) 

Please  describe  any  areas  in  which  you  felt  more  attention  should  have  been  provided  to  the  material 
(within  the  course  time  constraints). 

•  Little  more  time  going  more  in-depth  with  clinical  applications. 

•  Working  with  stigma  and  resistance 

•  Since  I  am  a  civilian,  some  of  the  more  mundane  concepts,  e.g.,  rank  and  hierarchy,  specific 
protocols  in  relation  to  discipline  for  cause  and  "vernacular"  -  rank  for  each  branch  could  be  a 
"org  chart"  and  list  of  acronyms  would  be  helpful 

•  Use  all  service  members  taught  about  the  symptoms  of  mental  health  of  all  kind  and  call  on 
them  to  observe  each  other? 

•  Clinical  diagnostics,  i.e.,  Hx  for  psych  testing. 

•  MST  issues  as  they  relate  to  PTSD.  Research  findings  on  treatment  modalities  effective  and 
treatments.  Results  of  outline  research  finding  for  treatment  of  PTSD. 

In  one  sentence  or  phrase,  what  is  the  single  most  important  thing  you  learned  in  this  training? 

•  There  are  some  important  differences  in  working  with  PTS  military  and  other  population  -  speak 
the  language. 

•  Specifics  of  treatments  for  PTSD 

•  To  not  be  afraid  of  the  Vet's  experience 

•  Clinical  techniques  working  with  anger  and  anxiety. 

•  Able  to  readily  and  directly  apply  concepts  in  dealing  with  student  veterans  as  an  advisor  (Please 
see  comments  for  examples) 

•  Viewing  PTSD  as  broken-heartedness 

•  review  and  reinforce  knowledge  and  {illegible} 

•  Updated  knowledge  and  technology,  tx  modalities. 
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•  The  overall  understanding  of  PTSD,  its  correlation  with  depression  and  introduction  to  possible  tx 
models. 

•  It  was  a  lot  of  great  information. 

•  Interrelated  social/variables  that  co-exist  with  PTSD  dx.  Multi  layer  symptoms  of  PTSD  with 
depression,  relationship  issue  etc. 


As  another  indicator  of  perceived  value  of  the  course,  participants  were  asked  to  indicate  the 
appropriate  cost  for  the  course.  Of  the  9  respondents  who  completed  this  question,  most  responded 
that  the  appropriate  fee  for  the  15  hour  course  would  be  between  $150-  200. 

Chart  1.  Responses  to  "I  would  consider  $ _ to  be  a  suitable  cost  for  this  course  as  a  Continuing 

Education  course." 


Utility  Judgments:  Implications  for  Clinical  Practice 

Six  items  assessing  participants'  beliefs  and  intentions  regarding  applying  course  concepts  to  practice 
were  examined  for  internal  reliability.  Results  indicated  that  five  of  the  six  items  were  appropriate  to 
combine  (via  averaging)  into  a  scale,  which  demonstrated  reliability  around  the  acceptable  level 
(Cronbach's  a=  .64).  The  item,  "Today's  training  made  me  feel  less  equipped  to  practice  effectively  with 
veterans  and  military  families  (Reverse  scored),"  was  not  included  in  the  utility  scale.  Overall,  the 
average  response  to  the  utility  judgments  scale  was  4.39,  out  of  a  possible  5,  (range:  4.06  to  4.69), 
indicating  a  strong  sense  of  intention  to  apply  course  concepts  to  practice  with  military  clients.  Mean 
response  by  item,  as  well  as  response  frequency  by  item  are  presented  in  Table  4. 
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Table  4.  Perceived  influence  of  Clinical  Practices  with  Servicemembers  and  Veterans  course  on  clinical 
practice  (n  =  16  unless  otherwise  noted). 


Item  (Mean  response) 

N 

Percent 

Q9.  This  course  made  me  aware  of  the 
differences  in  civilian  and  military  culture  with 
regard  to  behavioral  health  practice  issues.  (4.44) 

Neither  agree  nor  disagree 

1 

6.3 

Agree 

7 

43.8 

Strongly  agree 

8 

50.0 

Q10.  As  a  result  of  what  1  learned  in  this  course,  1 
can  think  of  two  concrete  strategies  1  can  use  in 
my  practice  with  veteran  clients.  (4.25) 

Strongly  disagree 

1 

6.3 

Neither  agree  nor  disagree 

1 

6.3 

Agree 

6 

37.5 

Strongly  agree 

8 

50.0 

Qll.  Participation  in  this  course  will  impact  how  1 
interact  with  my  military-related  clients.  (4.69) 

Agree 

5 

31.3 

Strongly  agree 

11 

68.8 

Q12.  As  a  result  of  this  training,  1  will  be  more 
confident  in  seeking  out  opportunities  to  work 
with  veterans  and  military  families.  (4.50) 

Neither  agree  nor  disagree 

2 

12.5 

Agree 

4 

25.0 

Strongly  agree 

10 

62.5 

Q13.  Today's  training  made  me  feel  less  equipped 
to  practice  effectively  with  veterans  and  military 
families  (R).*  (4.75) 

Neither  agree  nor  disagree 

1 

6.3 

Agree 

2 

12.5 

Strongly  agree 

13 

81.3 

Q  14.  After  this  course,  1  am  confident  about 
where  to  look  for  additional  resources  and 
information  on  working  with  veteran  and  military 
family  clients.  (4.06) 

Strongly  disagree 

1 

6.3 

Agree 

11 

68.8 

Strongly  agree 

4 

25.0 
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Overall,  responses  to  utility  items  indicate  that  participants  indicated  that  the  course  would  influence 
how  they  interacted  with  military  clients  and  left  them  feeling  more  equipped  to  practice  effectively 
with  veterans  and  military  families.  The  item  in  this  section  receiving  the  least  strong  endorsement 
inquired  about  participants'  confidence  finding  additional  resources  and  information  on  working  with 
veterans  and  military  families.  Frequently  noted  in  the  training  literature  is  the  importance  of  a  trainee's 
opportunity  to  use  techniques  learned  in  the  practice  setting.  Thus,  an  open-ended  question  asked 
participants  about  any  potential  barriers  to  applying  what  they  learned  in  their  own  practice.  Taken 
together,  responses  suggest  a  desire  to  gain  more  knowledge  about  military  culture  as  well  as  to 
develop  skills  in  appropriate  treatment  methods.  See  below  for  participants'  responses. 

In  a  sentence  or  two,  what,  if  any,  barriers  do  you  anticipate  could  prevent  you  from  using  the 
knowledge  you  acquired  today  in  your  practice? 

•  My  own  blindness  to  and  on  failure  to  ask  re  military  service. 

•  Working  with  co-occurring,  homeless,  OIF/OEF  vets,  there  is  a  lot  of  stigma  we  have  to  dig 
through  to  implement  some  of  what  was  taught. 

•  Being  on  the  resource  as  a  professional  -  to  get  appropriate  referrals. 

•  There  was  some  info  that  lacked  depth  and  clarity  that  would  have  seemed  more  useful  to  me.  It 
seemed  it  may  have  had  to  do  with  lack  of  time  allotted. 

•  /  think  I  need  more  training  to  get  to  know  the  culture  here  in  order  to  prepare  to  work  with  this 
population. 

•  Finding  certification/learning  to  be  skilled  in  various  treatment  modalities  to  incorporate  in 
treatment. 


Level  2:  Learning 

Level  2  of  the  Kirkpatrick  Model,  Learning  criteria,  refers  to  the  knowledge  acquired,  skills  improved,  or 
attitudes  changed  due  to  training  (Kirkpatrick,  1996).  Learning  criteria  are  internal  to  the  training  and 
are  typically  measured  immediately  after  the  training  occurs.  Because  the  Clinical  Practices  with 
Servicemembers  and  Veterans  CEU  course  sought  to  introduce  participants  to  military  life  experiences 
and  related  stress  reactions  that  practitioners  need  to  know  about  in  order  to  practice  effectively,  a 
pre/post  test  of  course  content  knowledge  was  used  as  a  measure  of  learning. 

Pre/Post  Knowledge  Gains 

A  20-item  multiple  choice  &  true/false  measure  assessed  participant  gains  in  knowledge  regarding 
clinical  practice  with  servicemembers  and  veterans  as  a  result  of  the  15-hour  course.  Results  indicated 
that  overall,  participants  scored  higher  on  the  knowledge  measure  at  the  conclusion  of  the  course  (M  = 
14.93,  SD  =  2.09)  than  at  the  beginning  of  the  course  (M  =  12.93,  SD  =  2.40),  t(15)  =  3.87,  p  <  .05. 
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Between  Group  Comparisons 

Between  group  comparisons  were  explored  using  a  series  of  independent  sample  t-tests.  Dependent 
variables  of  interest  included:  responses  to  affective  measures,  responses  to  utility  measures,  and 
pre/posttest  knowledge  gains.  No  significant  differences  emerged  for  these  outcomes  based  on: 
professional  discipline  (i.e.,  MFT,  Social  Worker),  years  practicing  (i.e.,  more  than  10,  less  than  10),  or 
amount  of  practice  spent  with  military-related  clients  (i.e.,  most  practice  with  military-clients,  very  little 
practice  with  military  clients),  all p  >  .05.  See  Table  5  for  means  and  standard  deviations. 

Table  5.  Group  means  and  standard  deviations  for  Affect,  Utility,  and  Knowledge  Measures. 


Affect  Mean  (SD) 

Utility  Mean  (SD) 

Pre/post 
knowledge  gain 
(SD) 

Discipline 

Social  Work 

4.32  (.39) 

4.32  (.41) 

1.00  (1.41) 

MFT 

4.69  (.24) 

4.40  (.45) 

2.25  (2.49) 

Years  Practicing 

<  10 

4.58  (.23) 

4.37  (.53) 

2.83  (2.14) 

>10 

4.48  (.44) 

4.44  (.52) 

1.44  (1.81) 

Practice  with 

Military  Clients 

Most 

4.56  (.27) 

4.25  (.60) 

1.00  (1.00) 

Very  little 

4.50  (.42) 

4.43  (.50) 

2.25  (2.14) 

Summary 

Available  data  on  Levels  1  and  2  of  the  Kirkpatrick  Training  Evaluation  Model  suggest  that  participants 
were  highly  satisfied  with  the  course,  believed  knowledge  gained  in  the  course  will  affect  how  they 
approach  clinical  practice  with  veterans  and  military  families,  and  demonstrated  gains  in  knowledge 
regarding  clinical  practice  with  servicemembers  and  veterans  over  the  15  hours  of  instruction.  No 
statistically  significant  differences  emerged  on  these  three  domains  by  profession,  experience,  or 
amount  of  practice  dedicated  to  military  clients.  With  regard  to  refining  the  course,  survey  results 
indicated  that  course  instruction  should  be  more  balanced  across  modalities  (e.g.,  small  group  activities, 
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discussion).  In  addition,  participants  do  not  report  high  confidence  in  knowing  where  to  turn  for  further 
resources  on  practice  with  veteran  and  military  family  clients. 

Next  Steps 

Level  3:  Behavior,  Level  4:  Results 

The  third  and  fourth  levels  of  the  Kirkpatrick  Evaluation  Model  relate  to  behavior  changes  in  the 
individual  and  ultimately,  the  organization,  as  a  result  of  training.  Outcomes  in  this  level  are  measured 
at  follow-up.  Please  see  Evaluation  Report  1  for  a  full  discussion. 

Recommendations 

•  Urge  instructors  to  plan  a  variety  of  instruction  methods  (small  groups,  interactive  activities  & 
exercises)  within  the  2  course  days. 

•  Inform  course  participants  of  resources  they  can  turn  to  for  further  information  on  the  topics 
covered  throughout  instruction,  in  addition  to  the  resource  binder  provided  to  them. 

•  To  the  extent  possible,  provide  basic  handouts  on  military  culture  (e.g.,  common  acronyms)  at 
each  course.  Some  participants  who  did  not  participate  in  the  Military  Culture  course  assumed 
they  had  missed  a  key  glossary  of  military  acronyms  from  that  course.  This  was  not  the  case,  but 
providing  this  information  to  all  students  may  help  them  feel  more  comfortable  that  they  have 
all  the  information  available  to  all  students. 
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Evaluation  Findings:  Continuing  Education  Course  3 
Families  Impacted  by  Military  Service:  Understanding  and  Intervening 

Twenty-two  behavioral  health  providers  participated  in  the  third  continuing  education  course  offered  by  CIR; 
Families  Impacted  By  Military  Service:  Understanding  and  Intervening.  Participants  were  surveyed  through  a 
multidimensional  questionnaire,  which  addressed  satisfaction,  perceptions  of  course  content  and  delivery,  and 
implications  for  professional  self  efficacy  and  clinical  practice.  Twenty-four  open-  and  closed-ended  questions, 
as  well  as  items  measuring  the  demographic  profile  of  respondents,  comprised  the  evaluation  measure. 

Participant  Characteristics 

Table  1  displays  demographic  breakdowns  of  the  participants.  Participants  were  relatively  equally  distributed 
with  regard  to  gender  (59%  female)  and  over  half  (55%)  were  over  age  55.  Nearly  three-quarters  (73%) 
identified  as  White.  Approximately  half  of  participants  (46%)  were  Marriage  and  Family  Therapists  with  a  third 
(27%)  identifying  as  Social  Workers.  Other  individuals  identified  themselves  as  psychologists,  psychiatrists, 
nurses  and  nurse  practitioners. 

With  regard  to  professional  qualifications,  two-thirds  (68%)  are  currently  licensed  with  another  fifth  (18%) 
registered  as  Social  Work  Associates  or  Marriage  and  Family  Therapy  Interns.  Of  the  course  participants  over 
one-third  (35%)  are  relatively  new  clinicians,  reporting  less  than  five  years  of  postgraduate-level  clinical  practice, 
with  an  equal  amount  (35%)  reporting  over  20  years  experience.  Most  participants  (70%)  work  full-time  in 
behavioral  health,  representing  the  fields  of  academia,  aging,  child  welfare,  health,  mental  health, 
developmental  disabilities,  and  substance  abuse.  Most  participants  focus  all  or  most  of  their  time  on  direct 
practice  (59%)  and  spend  at  least  some  of  their  time  working  with  military-related  populations  (69%). 


Table  1.  Characteristics  of  participants  (n=22) 


Characteristic 

N 

Percentage 

Age 

25-34 

1 

4.5 

35-44 

5 

22.7 

45-54 

4 

18.2 

55-64 

10 

45.5 

65+ 

2 

9.1 

Gender 

Female 

13 

59.1 

Male 

9 

40.9 

Race  and  Ethnicity 

Asian/Pacific  Islander 

2 

9.1 

Black/African-American 

1 

4.5 

Hispanic/Latino(a) 

2 

9.1 

Anglo/White 

16 

72.7 

Other 

1 

4.5 

1 
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Professional  Discipline 

Marriage  &  Family  Therapist 

10 

45.5 

Psychologist 

1 

4.5 

Social  Worker 

6 

27.3 

Other  (Psychiatrist,  Nurse  Practitioner) 

3 

13.6 

Veteran  Status 

Veteran 

7 

31.8 

Non-veteran 

15 

68.2 

Licensure  Status 

Licensed 

15 

68.2 

In  process  (registered) 

4 

18.2 

Not  licensed 

3 

13.6 

Years  of  Postgraduate  Practice 

Less  than  5 

7 

35.0 

5-9 

1 

5.0 

10-14 

3 

15.0 

15-19 

2 

10.0 

20  or  more 

7 

35.0 

Employment  Status 

Full-time  behavioral  health 

14 

70.0 

Part-time  behavioral  health 

3 

15.0 

Employed,  non-  behavioral  health 

3 

15.0 

Practice  Setting(s)* 

Academia 

1 

- 

Aging 

1 

- 

Child  Welfare 

2 

- 

Health 

1 

- 

Mental  Health 

16 

- 

Substance  Abuse 

3 

- 

Percentage  of  Time  Spent  in  Direct  Practice 

All  or  nearly  all  (>90%) 

10 

45.5 

Most  (51-90%) 

3 

13.6 

Some  (11-50%) 

6 

27.3 

Little  (<10%) 

3 

13.6 

Percentage  of  Time  with  Military  Populations 

All 

5 

23.8 

Most  (51-90%) 

1 

4.8 

Some  (11-50%) 

8 

38.1 

Little  (<10%) 

7 

33.3 

*  Adds  to  more  than  22  cases,  as  selection  of  multiple  practice  settings  was  permitted 
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Evaluation  Model 

The  4-level  Kirkpatrick  Training  Evaluation  Model  is  used  as  a  framework  in  which  to  consider  CIR's  CEU  training 
series.  The  levels  of  the  Kirkpatrick  Model  are  described  below  in  Table  2.  Briefly,  the  model  posits  4  levels  of 
evaluation:  reaction  criteria,  learning  criteria,  behavior  criteria,  and  results  criteria  (Kirkpatrick,  1996;  Praslova, 
2010).  Reaction  and  learning  criteria  are  considered  internal  criteria,  as  they  focus  on  changes  that  occur  within 
the  training  program  itself.  Behavioral  and  results  criteria  are  considered  external  criteria  and  occur  after  the 
training;  behavioral  and  results  criteria  are  influenced  by  factors  beyond  the  scope  of  the  training  (e.g., 
organizational  context,  opportunity/support  to  use  and  develop  skills).  For  a  more  complete  description  of  the 
Kirkpatrick  model,  please  see  the  Evaluation  Report  1:  Military  Culture  Course. 

Table  2.  Kirkpatrick's  4  Levels  of  Training  Evaluation. 


Level 

Criteria 

Locus  of  Change 

Variable  Measured 

Timepoint 

Measured 

1 

Reaction 

Internal  to  training 

Affective  (satisfaction) 

&  Utility  (plans  to 
implement)  judgments 

At  training 

2 

Learning 

Internal  to  training 

Knowledge  &  skill 
gains 

At  training 

3 

Behavioral 

External  to  training 

On-the-job 

performance 

Follow-up 

4 

Results 

External  to  training 

Productivity  gains, 
organizational  change 

Follow-up 

Level  1:  Reaction  Criteria 

Reaction  criteria  consist  of  two  elements  internal  to  the  training  itself:  affective  reaction  and  utility  judgments. 
Affective  reaction  refers  to  participants'  satisfaction  and  enjoyment  of  the  course.  Utility  judgments  refer  to 
participants'  beliefs  about  how  much  they  learned  and  how  they  plan  to  use  what  they  learned  in  their  practices 
with  veterans  and  military  families. 


Reaction  criteria  were  assessed  via  nine  Likert-type  items  regarding  perceptions  of  course  content  and  delivery 
and  participants'  intentions  to  use  knowledge  learned  during  the  course  in  clinical  practice.  Each  item  was  rated 
on  a  scale  of  1  (Strongly  disagree)  to  5  ( Strongly  agree).  One  item  was  reverse  scored,  as  indicated  below. 
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Items  were  analyzed  in  the  context  of  two  scales  -  one  assessing  participants'  affective  reaction  to  the  course, 
that  is  how  the  course  met  with  their  expectations  in  terms  of  quality  of  instruction,  breadth  and  depth  of 
material,  and  general  satisfaction,  and  the  other  assessing  participants'  utility  judgments  about  the  course  -  that 
is,  participants'  beliefs  that  they  will  use  what  they  learned  in  clinical  practice  with  military-related  populations 
and  that  they  gained  agency  in  working  effectively  with  military  populations. 

Affective  Reaction:  Perceptions  of  Course  Content  and  Delivery 

Thirteen  items  assessing  satisfaction  with  the  course  content,  delivery,  and  overall  value  of  the  course  were 
combined  (via  averaging)  into  a  scale,  which  demonstrated  good  reliability  above  the  acceptable  level 
(Cronbach's  a=  .98).  Overall,  the  average  response  to  the  affective  reaction  scale  was  4.40,  out  of  a  possible  5, 
(range:  4.19  to  4.76),  indicating  a  high  level  of  satisfaction  with  the  course  content,  delivery,  and  overall  value. 
Mean  response  by  item,  as  well  as  response  frequency  by  item  are  presented  in  Table  3. 


Table  3.  Perceptions  of  course  content  and  delivery 


Item  (Mean  Response) 

N 

Percent 

Ql.  The  instructors  were  interesting  &  kept  my 

attention  throughout  the  training.  (4.43) 

Strongly  disagree 

1 

4.5 

Agree 

9 

40.9 

Strongly  agree 

12 

54.5 

Q2.  The  material  was  clearly  presented.  (4.38) 

Strongly  disagree 

1 

4.5 

Agree 

9 

40.9 

Strongly  agree 

12 

54.5 

Q3.  The  balance  of  instruction  methods  (lecture, 

discussion,  small  group  work)  was  about  right.  (4.19) 

Strongly  disagree 

1 

4.5 

Agree 

14 

63.6 

Strongly  agree 

7 

31.8 

Q4.  The  activities  and  exercises  aided  in  my  learning. 

(4.33) 

Strongly  disagree 

1 

4.5 

Neither  agree  nor  disagree 

1 

4.5 

Agree 

8 

36.4 

Strongly  agree 

12 

54.5 

Q5.  The  breadth  of  material  covered  was  appropriate. 

(4.38) 

Strongly  disagree 

1 

4.5 

Agree 

9 

40.9 

Strongly  agree 

12 

54.5 

Q6.  The  depth  of  the  material  covered  was 

appropriate.  (4.19) 

Strongly  disagree 

1 

4.5 

4 
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Disagree 

Agree 

Strongly  agree 

1 

11 

9 

4.5 

50.0 

40.9 

Q7.  This  course  was  delivered  at  an  appropriate  level 
for  practitioners  with  my  level  of  experience.  (4.29) 

Strongly  disagree 

1 

4.5 

Neither  agree  nor  disagree 

1 

4.5 

Agree 

10 

45.5 

Strongly  agree 

10 

45.5 

Q8.  As  a  behavioral  health  provider,  1  was  able  to 
relate  to  the  instructors'  perspectives.  (4.48) 

Strongly  disagree 

1 

4.5 

Agree 

8 

36.4 

Strongly  agree 

13 

59.1 

Q17.  The  course  covered  the  right  amount  of 
information  for  the  length  of  the  training.  (4.24) 

Strongly  disagree 

1 

4.8 

Neither  agree  nor  disagree 

2 

9.5 

Agree 

8 

38.1 

Strongly  disagree 

10 

47.6 

Q18.  The  topics  covered  were  what  1  was  hoping  to 
see  in  this  course.  (4.29) 

Strongly  disagree 

1 

4.8 

Neither  agree  nor  disagree 

1 

4.8 

Agree 

9 

42.9 

Strongly  agree 

10 

47.6 

Q21.  Overall,  the  course  was  valuable.  (4.52) 

Strongly  disagree 

1 

4.8 

Agree 

6 

28.6 

Strongly  agree 

14 

66.7 

Q22.  A  course  on  military  culture  should  be 
compulsory  for  practitioners  working  with  veterans 
and  military  families.  (4.76) 

Strongly  disagree 

1 

4.8 

Agree 

1 

4.8 

Strongly  agree 

19 

90.5 

Q23. 1  would  recommend  this  course  to  a  friend  or 
colleague  interested  in  working  with  veterans  and 
military  families.  (4.67) 

Strongly  disagree 

1 

4.8 

Agree 

3 

14.3 

Strongly  agree 

17 

81.0 
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As  shown  above,  items  with  the  most  favorable  responses  queried  about  recommending  the  course  to  a  friend 
or  colleague  and  the  importance  of  training  like  this  for  practitioners  working  with  veterans  or  military  families. 
Although  still  favorable,  the  item  with  the  least  strong  endorsement  was  related  to  the  depth  of  material 
presented.  Additional  information  about  participant  satisfaction  and  expectation  was  obtained  through 
participants'  responses  to  2  open-ended  questions.  Responses  to  those  items  are  presented  below: 

Please  describe  any  topics  that  you  expected  to  be  covered,  but  weren't. 

•  More  theory  and  more  full  representation  of  the  most  central  ESTs.  I  also  wish  that  some  of  the  existing 
intervention  models  were  presented  more  cautiously/critically. 

•  Was  great  except  that  if  the  1st  class  on  culture  is  a  pre-reg  for  this  one,  there  was  duplicative  culture 
info  in  this  one.  If  it's  w/o  pre-reg,  there  was  perfect  to  include. 

•  Cognitive  Behavior  Conjoint  Therapy 

•  DX  checklist  &  follow-up  data 

Please  describe  any  areas  in  which  you  felt  more  attention  should  have  been  provided  to  the  material  (within 
the  course  time  constraints). 

•  A  little  more  time  on  some  subjects  could  have  been  helpful-some  seemed  rushed  through  due  to  lack  of 
time 

•  A  diagram  of  the  soldier's  experience  i.e.  recruiter-basic  training-? 

•  Case  vignettes-assessment  &  approach  to  treatment 

•  This  course  was  well-balanced. 

•  More  in  depth  needed  on  intervention/therapeutic  technigues 

•  Theory!  As  both  researcher  &  practitioner  I  would  have  appreciated  discussion  of  cases  from  a  hypothesis 
testing  perspective. 

•  More  on  spouse  issues  perhaps  &  kids  issues  perhaps 

•  /  thought  it  would  have  been  better  to  have  chose  a  few  family  tx  methods  are  gone  deeper,  rather  than 
a  broad  brush  on  many  theories. 

•  All  great 

•  Sugar  coated- 1  think  the  info  was  to  delivered  in  a  very  "appropriate"  manner.  Appropriate  is  not  always 
key  to  building  relationships  within  military,  mostly  vets. 

•  Treatment  tools 

•  Breath  [sic] 

•  Perhaps  some  networking.  I  missed  1st  day-so  maybe  done  fine. 

In  one  sentence  or  phrase,  what  is  the  single  most  important  thing  you  learned  in  this  training? 

•  The  small  group  work  was  GREAT  for  application  of  principles  and  theory.  Often  adult  learners  are 
hesitant  but  it  always  "enlivened"  the  group.  Wonderful  dynamic. 

•  Interventional  processes 

•  The  communication  facilitation  sheet  that  Eugenia  gave  us  will  be  used  for  couples. 

•  Treatment  models  and  their  effectiveness 
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•  The  military  has  come  a  long  way  in  working  with  51/S  members,  then  families  and  recovery  is  possible. 

•  Look  at  veterans  as  healthy-  don 't  assume  PTSD  or  other  pathologies. 

•  Someone  cares  about  the  veterans 

•  As  I'm  not  trained  as  a  social  worker,  some  of  the  social  work  theories  were  interesting  to  me, 
particularly  those  associated  with  grief/loss.  Is  anyone  else  disturbed  that  the  comprehensive  Soldier 
Fitness  program  is  MANDATORY  for  all  personnel?  Or  that  it  hasn't  been  piloted? 

•  About  the  differences  in  experiences  and  thought  processes  of  military  personnel 

•  In  depth  about  mil  culture  &  support 

•  /  learned  a  lot  about  active-duty  issues  that  I  was  not  aware  of,  including  violence  in  families. 

•  PTSD  different  interventions 

•  The  family  models/modalities  for  conceptualization  and  intervention. 

•  Just  about  learning  about  military  families  as  a  whole. ..the  culture  &  families  what  approaches  are  being 
done  to  address  basically  everything. 

•  /  am  in  the  right  field 

•  To  understand  the  military  culture  and  be  able  to  treat  and  apply  techniques  that  were  provided  to  me. 

•  Better  grasp  of  military  as  an  individual  culture/ethnicity 

•  Other  modalities  of  tx 

•  All  was  good 

•  It  is  surprising  that  school  kid  of  military  families  have  such  poor  to  negligible  support  from  school 
community 


As  another  indicator  of  perceived  value  of  the  course,  participants  were  asked  to  indicate  the  appropriate  cost 
for  the  course.  Of  the  19  respondents  who  completed  this  question,  most  responded  that  the  appropriate  fee 
for  the  15  hour  course  would  be  between  $150-  200.  See  Chart  1. 

Chart  1.  Responses  to  "I  would  consider  $ _ to  be  a  suitable  cost  for  this  course  as  a  Continuing  Education 

course." 
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Utility  Judgments:  Implications  for  Clinical  Practice 

Six  items  assessing  participants'  beliefs  and  intentions  regarding  applying  course  concepts  to  practice  were 
examined  for  internal  reliability.  Results  indicated  that  the  six  items  were  appropriate  to  combine  (via  averaging) 
into  a  scale,  which  demonstrated  reliability  above  the  acceptable  level  (Cronbach's  a=  .85).  Overall,  the  average 
response  to  the  utility  judgments  scale  was  4.46,  out  of  a  possible  5  (range:  4.32  to  4.68),  indicating  a  strong 
sense  of  intention  to  apply  course  concepts  to  practice  with  military  clients.  Mean  response  by  item,  as  well  as 
response  frequency  by  item  are  presented  in  Table  4. 


Table  4.  Perceived  influence  of  course  on  clinical  practice 


Item  (Mean  response) 

N 

Percent 

Q9.  This  course  made  me  aware  of  the  differences  in 
civilian  and  military  culture  with  regard  to  behavioral 
health  practice  issues.  (4.45) 

Strongly  disagree 

1 

4.5 

Neither  agree  nor  disagree 

1 

4.5 

Agree 

6 

27.3 

Strongly  agree 

14 

63.6 

Q10.  As  a  result  of  what  1  learned  in  this  course,  1  can 
think  of  two  concrete  strategies  1  can  use  in  my 
practice  with  veteran  clients.  (4.45) 

Strongly  disagree 

1 

4.5 

Agree 

8 

36.4 

Strongly  agree 

13 

59.1 

Qll.  Participation  in  this  course  will  impact  how  1 
interact  with  my  military-related  clients.  (4.45) 

Strongly  disagree 

1 

4.5 

Neither  agree  nor  disagree 

1 

4.5 

Agree 

6 

27.3 

Strongly  agree 

14 

63.6 

Q12.  As  a  result  of  this  training,  1  will  be  more 
confident  in  seeking  out  opportunities  to  work  with 
veterans  and  military  families.  (4.32) 

Strongly  disagree 

1 

4.5 

Neither  agree  nor  disagree 

2 

9.1 

Agree 

7 

31.8 

Strongly  agree 

12 

54.5 

Q13.  Today's  training  made  me  feel  less  equipped  to 
practice  effectively  with  veterans  and  military  families 
(R).  (4.68) 

Strongly  disagree 

19 

86.4 

Disagree 

1 

4.5 

Neither  agree  nor  disagree 

1 

4.5 

Strongly  agree 

1 

4.5 
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Q 14.  After  this  course,  1  am  confident  about  where  to 
look  for  additional  resources  and  information  on 
working  with  veteran  and  military  family  clients. 

(4.41) 

Strongly  disagree 

1 

4.5 

Neither  agree  nor  disagree 

1 

4.5 

Agree 

7 

31.8 

Strongly  agree 

13 

59.1 

Overall,  responses  to  utility  items  indicate  that  participants  felt  that  the  course  would  influence  how  they 
interacted  with  military  clients  and  left  them  feeling  more  equipped  to  practice  effectively  with  veterans  and 
military  families.  The  item  in  this  section  receiving  the  least  strong  endorsement  inquired  about  participants' 
confidence  finding  additional  resources  and  information  on  working  with  veterans  and  military  families. 

Frequently  noted  in  the  training  literature  is  the  importance  a  trainee's  opportunity  to  use  techniques  learned  in 
the  actual  practice  setting.  Thus,  an  open-ended  question  asked  about  any  potential  barriers  to  applying  what 
participants  learned  in  their  own  practice.  Responses  indicate  that  several  course  participants  anticipated 
structural  barriers  (below,  in  bold)  within  their  practice  setting,  such  as  policy  regulations  and  services  offered 
by  their  organization  that  may  prevent  them  from  applying  what  they  learned  in  training. 

In  a  sentence  or  two,  what,  if  any,  barriers  do  you  anticipate  could  prevent  you  from  using  the  knowledge  you 
acquired  today  in  your  practice? 

•  include  a  little  more  explanation  of  the  concept  of  "fortress"  up  front-early  in  overview-  provider  a  very 
concrete,  contextual  example  of  family/environment  dynamic.  Were  there  some  overall  objective  by 
'day'?  I  may  have  missed  them  but  they  might  be  pasted  [sic],  A  soldier's  family  ethnicity/culture 
important  to  include  when  discussing  tradition  re:  to  loss,  death,  rituals,  customs. 

•  /  do  not  think  that  there  are  any  barriers.  I  learned  a  lot  here  that  will  help  me  be  more  effective  in  the 
future. 

•  lack  of  opportunity  to  work  with  vets  hinders  development  of  my  newly  aeguired  skill/knowledge 

•  none.  I  would  highly  recommend  this  training  to  colleagues.  Everyone  is  so  well  organized,  prepared,  and 
provide  practical  information  and  experience. 

•  Lack  of  funding  for  re  type  of  service  (home-based)  that  my  agency  provides. 

•  excellent  class 

•  /  am  uncertain  about  the  extent  to  which  each  of  the  strategies  shared  are  evidence-based,  and  I  have  a 
commitment  to  providing  EBTs  to  my  clients  whenever  possible,  and  without  adaptation  that  have  not 
been  subjected  to  empirical  scrutiny. 

•  As  a  physician  psychiatrist,  insurances  prevent  me  from  seeing  patients  frequently  as  required.  No  time 
for  therapy  besides  medications. 

•  Client's  own  defenses 

•  client  avoidance  is  a  major  problem  with  our  young  vets.  Also,  a  lack  of  a  solid  family  program  will  limit 
my  exposure  to  couple/family. 

•  if  I  wanted  further  training  in  a  particular  model;  its  availability  to  attend  a  course. 
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•  lack  of  resources 

•  /  am  limited  to  providing  housing  to  vets  &  case  management  services  &  must  refer  to  MH  for  that  TX 
services.  Not  allowed  to  provided  PTSD  TX. 

•  My  geographic  location  is  45  min  plus  drive  from  the  nearest  base  -re  families. 


Level  2:  Learning 

Level  2,  Learning  criteria,  of  the  Kirkpatrick  Model  refers  to  the  knowledge  acquired,  skills  improved,  or  attitudes 
changed  due  to  training  (Kirkpatrick,  1996).  Learning  criteria  are  internal  to  the  training  and  are  typically 
measured  immediately  after  the  training  occurs.  Because  the  Families  Impacted  by  Military  Service: 
Understanding  and  Intervening  CEU  course  sought  to  introduce  participants  to  the  military  lifestyle  as  a  culture 
in  which  a  successful  practitioner  will  gain  competence  in  order  to  practice  effectively,  a  pre/post  test  of 
military-related  knowledge  was  used  as  a  measure  of  learning. 

Pre/Post  Knowledge  Gains 

A  20-item  multiple  choice  &  true/false  measure  assessed  participant  gains  in  knowledge  as  a  result  of  the  15- 
hour  course.  Results  indicated  that  overall,  participants  scored  higher  on  the  knowledge  measure  at  the 
conclusion  of  the  course  (65%  correct;  M  =  13.00,  SD  =  2.40)  than  at  the  beginning  of  the  course  (58%  correct;  M 
=  11.58,  SD  =  2.43),  t(19)  =  2.48,  p  <  .05.  These  differences  are  statistically  significant,  but  the  extent  to  which 
they  are  practically  meaningful  remains  an  open  questions.  However,  it  should  be  noted  that  considered  across 
the  entire  series,  knowledge  scores  at  pretest  have  increased  across  the  3  courses.  Although  each  course  is 
designed  to  stand  alone,  the  series  of  courses  are  complementary  and  build  upon  one  another.  In  other  words, 
someone  attending  all  the  courses  would  be  expected  to  have  an  increased  knowledge  of  military  culture  and 
practice  issues,  which  would  increase  pretest  knowledge  scores.  Seventeen  of  the  twenty-two  participants  had 
attended  one  or  two  of  the  other  CIR  courses  offered  in  the  CEU  series  previous  to  attending  the  Military 
Families  course. 

Between  Group  Comparisons 

Between  group  comparisons  were  explored  using  a  series  of  independent  sample  f-tests.  Dependent  variables  of 
interest  included:  responses  to  affective  measures,  responses  to  utility  measures,  and  pre/posttest  knowledge 
gains.  No  significant  differences  emerged  for  these  outcomes  based  on:  professional  discipline  (i.e.,  MFT,  Social 
Worker),  years  practicing  (i.e.,  more  than  10,  less  than  10),  or  amount  of  practice  spent  with  military-related 
clients1  (i.e.,  most  practice  with  military-clients,  very  little  practice  with  military  clients).  Although  not 
statistically  significant,  some  trends  emerged  such  that  social  workers  demonstrated  larger  gains  in  knowledge 
compared  to  MFT  participants  and  participants  with  at  least  ten  years'  experience  reported  greater  satisfaction 
with  the  course  than  participants  with  less  than  10  years'  experience.  See  Table  5  for  means  and  standard 
deviations. 
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Table  5.  Group  means  and  standard  deviations  for  Knowledge,  Affect,  and  Utility  Measures. 


Pre/post  Knowledge 
Gain  (SD) 

Affect  Mean  (SD) 

Utility  Mean  (SD) 

Discipline 

Social  Work 

3.00(2.55) 

4.03  (1.49) 

4.22  (1.26) 

MFT 

0.38  (2.26) 

4.48  (0.36) 

4.52  (0.32) 

Years  Practicing 

Less  than  10  years 

1.17  (2.93) 

3.96  (1.49) 

4.14  (1.16) 

10  +  years 

1.20(2.66) 

4.63  (0.21) 

4.70  (0.30) 

Practice  with 

Military  Clients 

All  or  most  (51%  +) 

1.50(2.38) 

3.92  (1.66) 

4.08  (1.25) 

Some  (11-50%) 

1.38  (2.72) 

4.53  (0.35) 

4.65  (0.30) 

Little 

0.83  (2.40) 

4.53  (0.46) 

4.50  (0.41) 

Veteran  Status 

Veteran 

1.17  (2.11) 

4.71  (0.20) 

4.73  (0.27) 

Non-veteran 

1.54(3.43) 

4.24(1.01) 

4.34  (0.83) 

1  Because  participants  were  approximately  evenly  distributed  into  3  groups,  a  one-way  ANOVA  was  used  to  examine 
differences. 


Summary 

In  conclusion,  results  from  the  first,  on-ground  pre-pilot  CEU  course  on  Families  Impacted  by  Military  Service  are 
promising.  Available  data  on  Levels  1  and  2  of  the  Kirkpatrick  Training  Evaluation  Model  suggest  that 
participants  were  highly  satisfied  with  the  course,  participants  believe  knowledge  gained  in  the  course  will  affect 
how  they  approach  clinical  practice  with  veterans  and  military  families,  and  participants  demonstrated  gains  in 
knowledge  over  the  2  days  of  instruction.  No  statistically  significant  differences  emerged  on  these  by  profession, 
experience,  or  amount  of  practice  dedicated  to  military  clients.  With  regard  to  refining  the  course,  participants 
requested  that  knowledge  be  presented  at  a  more  basic  level  and  that  more  interactive  training  modalities  be 
used  in  instruction. 
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Next  Steps 


Level  3:  Behavior 

The  third  level  of  the  Kirkpatrick  Evaluation  Model  measures  the  extent  participants  change  their  on-the-job 
behavior  as  a  result  of  the  training  (Kirkpatrick,  1996).  The  use  of  training  knowledge  and  skills  is  commonly 
referred  to  as  transfer  of  training.  Transfer  is  considered  an  external  criteria  and  is  frequently  influenced  by 
opportunities  within  one's  work  setting  to  practice  new  skills  (Clarke,  2002). 

As  transfer  is  conceptualized  to  occur  after  the  training,  it  is  best  measured  at  follow-up. 

To  assess  the  extent  to  which  participants  in  the  Families  Impacted  by  Military  Service:  Understanding  and 
Intervening  CEU  course  transfer  new  knowledge  and  skills  to  their  clinical  practices,  it  is  recommended  that 
participants  be  queried  via  internet  survey  at  3  and  6  months  post  course  participation  (i.e.,  early  June  2011, 
early  September  2011  respectively).  Following  Clarke  (2002),  domains  on  which  transfer  may  be  assessed  are: 

•  participants'  beliefs  about  the  benefits  gained  as  a  result  of  the  course 

•  evidence  of  using  the  training  in  practice  with  military-related  clients 

•  barriers  to  implementing  knowledge  and  skills  learned  in  the  training 

•  factors  that  facilitated  transfer  to  practice  with  military-clients 

At  follow-up,  participants  could  also  be  asked  to  review  their  records  for  the  last  year  to  provide  typical  attrition 
rates  for  military-related  clients  for  the  6  months  prior  to,  and  the  6  months  following,  their  participation  in  the 
Families  Impacted  by  Military  Service:  Understanding  and  Intervening  CEU  course. 


Level  4:  Results 

Results  criteria  refer  to  the  final  results  that  occur  due  to  training  (Kirkpatrick,  1996)  and  extend  beyond  the 
individual  participant  in  training  to  broader  organizational  change  (e.g.,  productivity  gains,  increased  employee 
morale)  (Praslova,  2010).  Results  criteria  are  used  less  frequently  than  the  other  levels  of  the  Kirkpatrick  model 
as  they  are  difficult  to  evaluate  (Praslova,  2010). 

In  the  current  evaluation,  results  criteria  can  be  measured  in  a  number  of  ways.  At  the  3  and  6  month  follow-up, 
participants  can  be  queried  the  extent  to  which: 

•  they  have  changed  the  way  they  train  other  clinicians  in  their  practice  who  work  with  military-related 
clients 

•  training  requirements  for  clinicians  in  their  organization  who  work  with  military-related  clients  have 
changed 

•  information  provided  in  the  course  (e.g.,  binder  resources)  facilitates  transfer  in  their  practice 
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•  attrition  rates  have  changed  for  military-related  clients  for  the  6  months  prior  to,  and  the  6  months 
following,  their  participation  in  the  Families  Impacted  by  Military  Service:  Understanding  and 
Intervening  CEU  course 

•  client-to-client  referral  rates  for  military-related  clients  have  changed  for  the  6  months  prior  to,  and  the 
6  months  following,  their  participation  in  the  Families  Impacted  by  Military  Service:  Understanding  and 
Intervening  CEU  course 

Results  criteria  can  be  measured  in  more  depth  in  the  Randomized  Controlled  Trial  (RCT)  planned  for  2012. 

Recommendations 

•  From  a  community-capacity  building  and  workforce-development  perspective,  instructors  should 
consider  devoting  time  in  the  course  to  coaching  participants  on  how  to  find  resources  relevant  to 
military  family  clients,  perhaps  as  an  interactive  or  small  group  activity. 

•  From  an  evaluation  perspective,  a  useful  item  on  the  survey  would  be  how  many  courses  they  have 
previously  taken  in  the  series.  Currently,  it  is  possible  to  determine  what  overall  portion  of  participants 
have  attended  a  previous  course  in  the  series  (based  on  course  sign-in  sheets),  but  this  cannot  be  linked 
with  their  survey  responses.  Linking  previous  participation  with  knowledge  scores  would  allow  us  to 
better  characterize  knowledge  gains  on  the  pre  test  over  the  entire  course  series. 
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Evaluation  Findings:  Continuing  Education  Course  4 
Health  Challenges  for  Wounded  Warriors  and  their  Caregivers 

Twenty-five  behavioral  health  providers  participated  in  the  fourth  continuing  education  course  offered 
by  CIR;  Health  Challenges  for  Wounded  Warriors  and  their  Caregivers.  Participants  were  surveyed 
through  a  multidimensional  questionnaire,  addressing  satisfaction,  perceptions  of  course  content  and 
delivery,  and  implications  for  clinical  practice.  The  evaluation  measure  consisted  of  19  Likert-type  items, 
5  open-ended  items,  and  12  demographic  items. 

Participant  Characteristics 

As  shown  in  Table  1,  participants  were  predominantly  female  (70%)  and  relatively  evenly  distributed 
with  regard  to  age.  In  fact,  over  half  (58%)  were  age  55  and  over.  Just  over  half  of  the  course 
participants  (57%)  identified  as  White.  Participants  were  predominantly  Marriage  and  Family  Therapists 
(39.1%)  and  Social  Workers  (47.8%).  Nearly  one-fifth  (17%)  of  participants  identified  as  a  current  or 
former  servicemember,  with  nearly  half  (40%)  reporting  having  a  servicemember  in  their  family. 

With  regard  to  professional  qualifications,  most  (70%)  were  currently  licensed,  with  the  remainder 
(30%)  currently  working  toward  licensure  (i.e.,  registered  as  Social  Work  Associates  or  Marriage  and 
Family  Therapy  Interns).  One-quarter  (26%)  of  participants  were  relatively  new  clinicians,  reporting  less 
than  five  years  of  postgraduate-level  clinical  practice,  with  nearly  half  (48%)  reporting  over  20  years 
experience.  Most  participants  (75%)  worked  full-time  in  behavioral  health.  Three-quarters  of 
participants  reported  (76%)  spending  most  of  their  time  in  direct  practice.  The  majority  of  participants 
(59%)  indicated  spending  10%  or  more  of  their  time  working  with  military-related  populations. 


Table  1.  Characteristics  of  participants  (n=23,  who  completed  demographic  measures) 


Characteristic 

N 

Percent 

Age 

<25 

1 

4.3 

25-34 

5 

21.7 

35-44 

1 

4.3 

45-54 

6 

21.7 

55-64 

8 

34.8 

65+ 

3 

13.0 

Gender 

Female 

16 

69.6 

Male 

7 

30.4 

Race  and  Ethnicity 

Asian/Pacific  Islander 

2 

8.7 

Black/African-American 

3 

13.0 

Hispanic/Latino(a) 

5 

21.7 

Anglo/White 

13 

56.5 
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Professional  Discipline 

Marriage  &  Family  Therapist 

9 

39.1 

Psychologist 

1 

4.3 

Social  Worker 

11 

47.8 

Other  (Psychiatrist,  Nurse  Practitioner) 

2 

8.7 

Veteran  Status 

Veteran 

4 

17.4 

Non-veteran 

19 

82.6 

Family  Service  Status 

Servicemember  in  family 

9 

40.9 

No  servicemembers  in  family 

14 

59.1 

Relationship  to  Servicemember 

Spouse 

2 

8.7 

Child 

1 

4.3 

Sibling 

4 

17.4 

Other  (e.g.,  parent,  niece/nephew) 

4 

17.4 

Licensure  Status 

Licensed 

16 

69.6 

In  process  (registered) 

7 

30.4 

Years  of  Postgraduate  Practice 

Less  than  5 

6 

26.1 

5-9 

3 

13.0 

10-14 

2 

8.7 

15-19 

1 

4.3 

20  or  more 

11 

47.8 

Employment  Status 

Full-time  behavioral  health 

15 

75 

Part-time  behavioral  health 

2 

10 

Employed,  non-  behavioral  health 

2 

10 

Not  currently  employed 

1 

5 

Practice  Setting(s)* 

Academia 

3 

- 

Aging 

4 

- 

Child  Welfare 

2 

- 

Health 

3 

- 

Mental  Health 

14 

- 

Substance  Abuse 

4 

- 

Percentage  of  Time  Spent  in  Direct  Practice 

All  or  nearly  all  (>90%) 

8 

38.1 

Most  (51-90%) 

8 

38.1 

Some  (11-50%) 

5 

23.8 

Percentage  of  Time  with  Military  Populations 

All 

4 

18.2 

Most  (51-90%) 

3 

13.6 

Some  (11-50%) 

6 

27.3 

Little  (<10%) 

9 

40.9 

*  Adds  to  more  than  23  cases,  as  selection  of  multiple  practice  settings  was  permitted 
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Evaluation  Model 

The  4-level  Kirkpatrick  Training  Evaluation  Model  is  used  as  a  framework  in  which  to  consider  CIR's 
Continuing  Education  (CE)  training  series.  The  levels  of  the  Kirkpatrick  Model  are  described  below  in 
Table  2.  Briefly,  the  model  posits  4  levels  of  evaluation:  reaction  criteria,  learning  criteria,  behavior 
criteria,  and  results  criteria  (Kirkpatrick,  1996;  Praslova,  2010).  Reaction  and  learning  criteria  are 
considered  internal  criteria,  as  they  focus  on  changes  that  occur  within  the  training  program  itself. 
Behavioral  and  results  criteria  are  considered  external  criteria  and  occur  after  the  training;  behavioral 
and  results  criteria  are  influenced  by  factors  beyond  the  scope  of  the  training  (e.g.,  organizational 
context,  opportunity/support  to  use  and  develop  skills).  For  a  more  complete  description  of  the 
Kirkpatrick  model,  please  see  Evaluation  Report  1:  Military  Culture. 


Table  2.  Kirkpatrick's  4  Levels  of  Training  Evaluation. 


Level 

Criteria 

Locus  of  Change 

Variable  Measured 

Timepoint 

Measured 

1 

Reaction 

Internal  to  training 

Affective  (satisfaction) 

&  Utility  (plans  to 
implement)  judgments 

At  training 

2 

Learning 

Internal  to  training 

Knowledge  &  skill 
gains 

At  training 

3 

Behavioral 

External  to  training 

On-the-job 

performance 

Follow-up 

4 

Results 

External  to  training 

Productivity  gains, 
organizational  change 

Follow-up 

Level  1:  Reaction  Criteria 

Reaction  criteria  consist  of  two  elements  internal  to  the  training  itself:  affective  reaction  and  utility 
judgments.  Affective  reaction  refers  to  participants'  satisfaction  and  enjoyment  of  the  course.  Utility 
judgments  refer  to  participants'  beliefs  about  how  much  they  learned  and  how  they  plan  to  use  what 
they  learned  in  their  practice  with  veterans  and  military  families. 


Reaction  criteria  were  assessed  through  nineteen  Likert-type  items  regarding  perceptions  of  course 
content  and  delivery  and  participants'  intentions  to  use  knowledge  learned  during  the  course  in  clinical 
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practice.  Each  item  was  rated  on  a  scale  of  1  (Strongly  disagree)  to  5  (Strongly  agree).  One  item  was 
reverse  scored,  as  indicated  below. 

Items  were  analyzed  in  the  context  of  two  scales.  One  assessed  participants'  affective  reaction  to  the 
course  -  that  is,  how  the  course  met  with  their  expectations  in  terms  of  quality  of  instruction,  breadth 
and  depth  of  material,  and  general  satisfaction.  The  other  examined  participants'  utility  judgments 
about  the  course  -  that  is,  participants'  beliefs  that  they  would  use  what  they  learned  in  clinical  practice 
with  military-related  populations  and  that  they  gained  agency  in  working  effectively  with  military 
populations. 


Affective  Reaction:  Perceptions  of  Course  Content  and  Delivery 

Thirteen  items  assessing  satisfaction  with  the  course  content,  delivery,  and  overall  value  of  the  course 
were  combined  (via  averaging)  into  a  scale,  which  demonstrated  high  reliability  (Cronbach's  a=  .90). 
Overall,  the  average  response  to  the  affective  reaction  scale  was  4.56  out  of  a  possible  5  (range:  4.19  to 
4.91),  indicating  a  high  level  of  satisfaction  with  the  course  content,  delivery,  and  overall  value.  Mean 
response  by  item,  as  well  as  response  frequency  by  item,  are  presented  in  Table  3. 


Table  3.  Perceptions  of  course  content  and  delivery 


Item  (Mean  Response) 

Frequency 

Percent 

Ql.  The  instructors  were  interesting  &  kept  my 
attention  throughout  the  training.  (4.46) 

Agree 

13 

54.2 

Strongly  agree 

11 

45.8 

Q2.  The  material  was  clearly  presented.  (4.46) 

Agree 

13 

54.2 

Strongly  agree 

11 

45.8 

Q3.  The  balance  of  instruction  methods  (lecture, 
discussion,  small  group  work)  was  about  right. 
(4.33) 

Neither  agree  nor  disagree 

1 

4.2 

Agree 

12 

58.3 

Strongly  agree 

12 

37.5 

Q4.  The  activities  and  exercises  aided  in  my 
learning.  (4.38) 

Agree 

15 

62.5 

Strongly  agree 

9 

37.5 

Q5.  The  breadth  of  material  covered  was 
appropriate.  (4.50) 

Agree 

12 

50.0 

Strongly  agree 

12 

50.0 
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Q6.  The  depth  of  the  material  covered  was 
appropriate.  (4.50) 

Neither  agree  nor  disagree 

1 

4.2 

Agree 

10 

41.7 

Strongly  agree 

13 

54.2 

Q7.  This  course  was  delivered  at  an  appropriate 
level  for  practitioners  with  my  level  of 
experience.  (4.54) 

Neither  agree  nor  disagree 

1 

4.2 

Agree 

9 

37.5 

Strongly  agree 

14 

58.3 

Q8.  As  a  behavioral  health  provider,  1  was  able  to 
relate  to  the  instructors'  perspectives.  (4.88) 

Agree 

7 

30.4 

Strongly  agree 

16 

69.6 

Q17.  The  course  covered  the  right  amount  of 
information  for  the  length  of  the  training.  (4.18) 

Disagree 

1 

4.5 

Neither  agree  nor  disagree 

1 

4.5 

Agree 

13 

59.1 

Strongly  disagree 

7 

31.8 

Q18.  The  topics  covered  were  what  1  was  hoping 
to  see  in  this  course.  (4.32) 

Neither  agree  nor  disagree 

1 

4.5 

Agree 

13 

59.1 

Strongly  agree 

8 

36.4 

Q21.  Overall,  the  course  was  valuable.  (4.91) 

Agree 

2 

9.1 

Strongly  agree 

20 

90.9 

Q22.  A  course  on  military  culture  should  be 
compulsory  for  practitioners  working  with 
veterans  and  military  families.  (4.91) 

Agree 

2 

9.1 

Strongly  agree 

22 

90.9 

Q23. 1  would  recommend  this  course  to  a  friend 
or  colleague  interested  in  working  with  veterans 
and  military  families.  (4.82) 

Agree 

4 

18.2 

Strongly  agree 

18 

81.8 

As  shown  above,  items  with  the  most  favorable  responses  inquired  about  the  overall  value  of  the  course 
and  the  importance  of  training  like  this  for  practitioners  working  with  veterans  or  military  families. 
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Although  still  favorable,  the  item  with  the  least  strong  endorsement  was  related  to  the  match  between 
practitioners'  experience  and  the  level  of  delivery  of  course  material.  Additional  information  about 
participant  satisfaction  and  expectation  was  obtained  through  participants'  responses  to  2  open-ended 
questions.  Responses  to  those  items  are  presented  below: 

Please  describe  any  topics  that  you  expected  to  be  covered,  but  weren't. 

•  Medications 

•  Secondary  Traumatic  Stress  Syndrome 

•  Different  cultural  challenges 

•  Military  Sexual  Trauma  &  its  relationship  to  PTSD,  and  what  can  be  done. 

•  More  information  addressing  female  VETS-  specific  issues  affecting  female  vets  -  distinguished 
from  men. 

•  The  handouts  are  a  bit  disorganized-maybe  have  materials  on  a  website  &  allow  us  to  follow  on 
own  computers. 

•  More  direct  practice  examples 

•  Person 's  sexual  trauma 

•  Prolonged  PTSD 


Please  describe  any  areas  in  which  you  felt  more  attention  should  have  been  provided  to  the  material 
(within  the  course  time  constraints). 

•  Therapies  related  to  psychiatric  problems 

•  Monica's  presentation  on  Friday  was  very  good  and  it  would  have  been  nice  to  have  more  time 
for  that  and  expand  on  some  topics. 

•  Providers  dealing  with  emotional  impact  working  with  the  population,  what  support  is  available 
to  the  providers. 

•  The  material  was  more  than  adequate 

•  Military  Sexual  Trauma  &  its  relationship  to  PTSD,  and  what  can  be  done. 

•  Having  all  handouts  ready  for  us  before  lecture. 

•  Change  speech  therapist  to  1st  day  so  we  can  better  understand  cognitive  issues  earlier  on. 

•  Therapeutic  Interventions  &  role  play 
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•  Excellent  comprehensive  coverage  of  subject-  looking  forward  to  becoming  more  familiar  with 
resource  identified. 

•  Women  in  military;  diagnostic  evaluations 

In  one  sentence  or  phrase,  what  is  the  single  most  important  thing  you  learned  in  this  training? 

•  Resource  website  information 

•  Roles  or  providers  in  TBI  clinic 

•  Resources  available 

•  More  resources  available. 

•  Web  resources 

•  All  the  resource  information 

•  More  effect/attention  is  needed  to  get  the  info  out  to  the  clinicians. 

•  Online  resources-where  to  go  for  VA/DOD  info 

•  That  I  have  a  lot  more  to  learn. 

•  Resources  =  really  ALL  of  it 

•  The  value  of  this  training  for  me  is  in  the  including  information  about  military  members  &  their 
families  in  my  teaching.  Additionally,  this  information  will  be  helpful  in  better  preparing  out 
students  for  placement  at  the  VA. 

•  Resources 

•  Warriors 

•  Most  of  the  general  info  validated  concepts  previously  learned 

•  Increased  understanding  of  some  of  the  common  health  challenges  of  our  wounded  warriors. 

•  Resource  DOD,  VA  etc.  sites 

•  A  lot  of  resources 

•  Directory  resources 

•  Support  and  know  where  to  look  for 
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•  David  Yi  Darla-  The  war  does  not  end  when  you  come  home. 

•  How  to  get  the  information  on  the  web. 


As  another  indicator  of  the  perceived  value  of  the  course,  participants  were  asked  to  indicate  the 
appropriate  cost  for  the  course.  Of  the  19  respondents  who  completed  this  open-ended  question,  the 
mode  response  was  between  $200  -  299  (42%).  See  Chart  1  below. 

Chart  1.  Responses  to  "I  would  consider  $ _ to  be  a  suitable  cost  for  this  course  as  a  Continuing 

Education  course." 


Utility  Judgments:  Implications  for  Clinical  Practice 

Six  items  assessing  participants'  beliefs  and  intentions  regarding  applying  course  concepts  to  practice 
were  examined  for  internal  reliability.  Results  indicated  that  five  of  the  six  items  were  appropriate  to 
combine  (via  averaging)  into  a  scale,  which  demonstrated  adequate  reliability  (Cronbach's  a=  .73). 
Overall,  the  average  response  to  the  utility  judgments  scale  was  4.57  out  of  a  possible  5  (range:  4.26  to 
4.74),  indicating  a  strong  sense  of  intention  to  apply  course  concepts  to  practice  with  military  clients. 
Mean  response  by  item,  as  well  as  response  frequency  by  item,  are  presented  in  Table  4. 
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Table  4.  Perceived  influence  of  course  on  clinical  practice 


Item  (Mean  response) 

N 

Percent 

Q9.  This  course  made  me  aware  of  the 
differences  in  civilian  and  military  culture  with 
regard  to  behavioral  health  practice  issues.  (4.25) 

Neither  agree  nor  disagree 

3 

12.5 

Agree 

12 

50.0 

Strongly  agree 

9 

37.5 

Q10.  As  a  result  of  what  1  learned  in  this  course,  1 
can  think  of  two  concrete  strategies  1  can  use  in 
my  practice  with  veteran  clients.  (4.50) 

Agree 

12 

50.0 

Strongly  agree 

12 

50.0 

Qll.  Participation  in  this  course  will  impact  how  1 
interact  with  my  military-related  clients.  (4.65) 

Agree 

8 

34.8 

Strongly  agree 

15 

65.2 

Q12.  As  a  result  of  this  training,  1  will  be  more 
confident  in  seeking  out  opportunities  to  work 
with  veterans  and  military  families.  (4.70) 

Neither  agree  nor  disagree 

1 

4.3 

Agree 

5 

21.7 

Strongly  agree 

17 

73.9 

Q13.  Today's  training  made  me  feel  less  equipped 
to  practice  effectively  with  veterans  and  military 
families  (R).*  (4.54) 

Strongly  disagree 

16 

66.7 

Disagree 

7 

29.2 

Strongly  agree 

1 

4.0 

Q  14.  After  this  course,  1  am  confident  about 
where  to  look  for  additional  resources  and 
information  on  working  with  veteran  and  military 
family  clients.  (4.75) 

Agree 

6 

25.0 

Strongly  agree 

18 

75.0 

*ltem  reverse  scored  for  analyses  so  that  all  items  are  coded  in  the  same  positive/negative  direction. 


Overall,  responses  to  utility  items  indicate  that  participants  felt  that  the  course  would  influence  their 
practice  with  military-related  clients.  The  item  with  the  strongest  endorsement  inquired  about 
participants'  confidence  finding  additional  resources  and  information  on  working  with  military-related 
clients.  This  indicates  an  improvement  over  reactions  to  previous  courses  in  the  training  series.  As 
noted  in  Evaluation  Reports  1-3,  participants  were  not  always  as  strong  on  this  item.  One  instructor  for 
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the  current  course  incorporated  a  significant  amount  of  interactive  activities  focused  on  finding 
additional  resources  online.  Although  still  favorable,  the  item  in  this  section  receiving  the  least  strong 
endorsement  asked  whether  participants  felt  more  aware  of  differences  between  civilian  and  military 
culture  with  regard  to  behavioral  health  practice  as  a  result  of  the  course. 

Frequently  noted  in  the  training  literature  is  the  importance  a  trainee's  opportunity  to  make  use  of 
newly-learned  techniques  in  the  actual  practice  setting.  Thus,  an  open-ended  question  asked  about 
potential  barriers  to  applying  what  participants  learned  in  their  own  practice.  Two  respondents 
mentioned  time  being  a  factor  to  implement  the  knowledge  gained  in  the  course  to  their  practice. 

In  a  sentence  or  two,  what,  if  any,  barriers  do  you  anticipate  could  prevent  you  from  using  the 
knowledge  you  acquired  today  in  your  practice? 

•  None  known 

•  Difficulty  staying  focused  during  presentation  due  to  medium  or  presentation  for  an  extended 
period  of  time. 

•  Great  class.  Want  more  classes. 

•  Access  to  copies  of  each  instructor's  PP  presentations 

•  None.  Excellent  course  over  all.  I  would  shorten  the  days  to  end  around  3:00-instead  of  4:30 

•  There  could  have  been  much  more  training  in  which  therapies  (mental)  work  in  PTSD/TBI  &  some 
examples  (videos)  of  actual  therapy  sessions. 

•  Limited  contact  w/  Vets  and/or  VA- 1  do  periodic  yellow  ribbon  event  have  Tricare,  is  referral,  but 
no  daily  contact  [sic] 

•  Time 

•  Oversight  of  persons  sexual  abuse-women  vets  in  the  shadows.  Folders  should  have  all  the  slides 
and  presentations. 

•  /  don 't  work  with  severely  physically  disabled  veterans  and  I  am  in  the  process  of  slowly 
incorporating  family/caregiver  support  in  my  program.  Lack  of  family  caregiver  participation. 

•  Time! 


Level  2:  Learning 

Level  2  of  the  Kirkpatrick  Model,  Learning  criteria,  refers  to  the  knowledge  acquired,  skills  improved,  or 
attitudes  changed  due  to  training  (Kirkpatrick,  1996).  Learning  criteria  are  internal  to  the  training  and 
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are  typically  measured  immediately  after  the  training  occurs.  Consistent  with  the  previous  3  CE  courses 
in  the  series,  a  pre/post  test  of  military-related  knowledge  was  used  as  a  measure  of  learning. 

Pre/Post  Knowledge  Gains 

A  20-item  multiple  choice  &  true/false  measure  assessed  participant  gains  in  knowledge  as  a  result  of 
the  15-hour  course.  Results  indicated  that  overall,  participants  scored  higher  on  the  knowledge  measure 
at  the  conclusion  of  the  course  (78%  correct;  M  =  15.61,  SD  =  1.94)  than  at  the  beginning  of  the  course 
(69%  correct;  M  =  13.72,  SD  =  2.02),  t(17)  =  4.59,  p  <  .0001. 

Considered  across  the  entire  series,  knowledge  scores  at  pretest  have  increased  throughout  the  4 
courses.  Although  each  course  is  designed  to  stand  alone,  the  series  of  courses  are  complementary  and 
build  upon  one  another.  Put  another  way,  someone  attending  all  the  courses  would  be  expected  to  have 
an  increased  knowledge  of  military  culture  and  practice  issues,  which  would  increase  pretest  knowledge 
scores.  Twelve  of  the  participants  had  attended  at  least  one  of  the  other  CIR  courses  offered  in  the  CE 
series  prior  to  attending  the  Health  Challenges  course. 

Between  Group  Comparisons 

Between-group  differences  were  explored  for  the  three  dependent  variables  of  interest:  responses  to 
affective  items,  responses  to  utility  items,  and  pre/posttest  knowledge  gains.  Independent  samples  t- 
tests  were  used  to  examine  between-group  differences  for  professional  discipline  (i.e.,  MFT,  Social 
Worker)  and  years  practicing  (i.e.,  more  than  10,  less  than  10).  No  statistically  significant  differences 
emerged  for  these  outcomes  based  on  professional  discipline  or  years  practicing. 

Because  participants  were  approximately  evenly  distributed  into  three  groups,  a  one-way  ANOVA  was 
used  to  examine  differences  on  the  three  dependent  variables  with  regard  to  practice  time  spent  with 
military  clients.  No  statistically  significant  differences  emerged  for  pre/posttest  knowledge  gains  or 
utility  items.  However,  there  was  a  significant  effect  regarding  the  portion  of  practice  time  spent  with 
military  clients  on  affective  reactions  to  the  course  ( F  (2,  17)  =  5.27,  p  <  .05).  Specifically,  Tukey  post-hoc 
comparisons  indicate  that  more  positive  affective  reactions  were  found  among  participants  who  spent 
either  (a)  all  or  most  of  their  time  with  military  clients  (M  =  4.68,  SD  =  .29)  or  (b)  some  of  their  time  with 
military  clients  (M  =  4.71,  SD  =  .27)  than  among  participants  who  spend  <  10%  of  their  time  with  military 
clients,  (M  =  4.24,  SD  =  .32).  Taken  together,  these  results  suggest  that  although  there  was  no  difference 
in  how  much  participants  learned  or  how  satisfied  participants  were  with  the  course,  practitioners  who 
spend  very  little  time  with  military  clients  did  not  anticipate  the  course  being  as  useful  to  their  practice 
as  those  who  spend  more  time  with  military  clients.  See  Table  5  for  means  and  standard  deviations. 
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Table  5.  Group  means  and  standard  deviations  for  Knowledge,  Affect,  and  Utility  Measures. 


Pre/post  Knowledge  Gain  (SD) 

Affect  Mean  (SD) 

Utility  Mean  (SD) 

Discipline 

Social  Work 

1.50  (1.07) 

4.33  (.35) 

4.62  (.40) 

MFT 

2.13  (2.42) 

4.57  (.36) 

4.45  (.38) 

Years  Practicing 

Less  than  10  years 

2.43  (2.23) 

4.61(.39) 

4.45  (.38) 

10  +  years 

1.40  (1.35) 

4.47(33) 

4.62  (.40) 

Practice  with 

Military  Clients 

All  or  most  (51%  +) 

1.20  (1.48) 

4.68  (.29) 

4.60  (.30) 

Some  (11-50%) 

3.25  (2.06) 

4.71  (.27) 

4.40  (.51) 

Little 

1.43  (1.72) 

4.24  (.32) 

4.63  (.38) 

Summary 

In  conclusion,  evaluation  results  from  the  first,  on-ground  pre-pilot  CE  course  on  Health  Challenges  for 
Wounded  Warriors  and  their  Caregivers  are  promising  and  informative.  Available  data  on  Levels  1  and  2 
of  the  Kirkpatrick  Training  Evaluation  Model  suggest  that  participants  were  highly  satisfied  with  the 
course,  believed  knowledge  gained  in  the  course  would  affect  how  they  approach  clinical  practice  with 
veterans  and  military  families,  and  demonstrated  gains  in  knowledge  over  the  2  days  of  instruction.  No 
statistically  significant  differences  emerged  on  these  by  profession,  experience,  or  amount  of  practice 
dedicated  to  military  clients. 


Next  Steps 


Level  3:  Behavior 

The  third  level  of  the  Kirkpatrick  Evaluation  Model  measures  the  extent  that  participants  change  their 
on-the-job  behavior  as  a  result  of  the  training  (Kirkpatrick,  1996).  The  use  of  training  knowledge  and 
skills  is  commonly  referred  to  as  transfer  of  training.  Transfer  is  considered  an  external  criterion  and  is 
frequently  influenced  by  opportunities  to  practice  new  skills  within  one's  work  setting  (Clarke,  2002).  As 
transfer  is  conceptualized  to  occur  after  the  training,  it  is  best  measured  at  follow-up. 

To  assess  the  extent  to  which  participants  in  the  Health  Challenges  CE  course  transfer  new  knowledge 
and  skills  to  their  clinical  practice,  it  is  recommended  that  participants  be  queried  via  internet  survey  at 
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3  and  6  months  post  course  participation  (i.e.,  early  June  2011,  early  September  2011  respectively). 
Following  Clarke  (2002),  domains  on  which  transfer  may  be  assessed  are: 

•  participants'  beliefs  about  the  benefits  gained  as  a  result  of  the  course 

•  evidence  of  using  the  training  in  practice  with  military-related  clients 

•  barriers  to  implementing  knowledge  and  skills  learned  in  the  training 

•  factors  that  facilitated  transfer  to  practice  with  military-clients,  including  which  elements  from 
the  course  were  most  helpful  -  such  as  resource  binder  materials,  etc. 

At  follow-up,  participants  could  also  be  asked  to  review  their  records  for  the  last  year  to  provide  typical 
attrition  rates  for  military-related  clients  for  the  six  months  prior  to,  and  the  six  months  following,  their 
participation  in  the  Health  Challenges  CE  course. 


Level  4:  Results 

Results  criteria  refer  to  the  final  results  that  occur  due  to  training  (Kirkpatrick,  1996)  and  extend  beyond 
the  individual  participant  in  training  to  broader  organizational  change  (e.g.,  productivity  gains,  increased 
employee  morale)  (Praslova,  2010).  Results  criteria  are  used  less  frequently  than  the  other  levels  of  the 
Kirkpatrick  model,  as  they  are  difficult  to  evaluate  (Praslova,  2010). 

In  the  current  evaluation,  results  criteria  can  be  measured  in  a  number  of  ways.  At  the  3-  and  6-month 
follow-up,  participants  can  be  queried  regarding  the  extent  to  which: 

•  they  have  changed  the  way  they  train  other  clinicians  in  their  practice  who  work  with  military- 
related  clients 

•  training  requirements  for  clinicians  in  their  organization  who  work  with  military-related  clients 
have  changed 

•  attrition  rates  have  changed  for  military-related  clients  for  the  6  months  prior  to,  and  the  6 
months  following,  their  participation  in  the  Health  Challenges  CE  course 

•  client-to-client  referral  rates  for  military-related  clients  have  changed  for  the  6  months  prior  to, 
and  the  6  months  following,  their  participation  in  the  Health  Challenges  CE  course 

Results  criteria  will  be  measured  in  more  depth  in  the  Randomized  Controlled  Trial  (RCT)  planned  for 
2012. 


Recommendations 

With  regard  to  course  content,  several  participants  indicated  wanting  to  learn  more  about  women  in  the 
military,  military  sexual  trauma,  and  compassion  fatigue.  CIR  is  currently  working  to  eliminate 
redundancies  between  courses  in  the  series.  These  topics  should  be  considered  either  as  additions  to 
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the  courses  once  current  redundancies  are  eliminated  or  as  potential  topics  for  an  additional  CE 
offering. 

For  course  delivery,  a  few  logistical  changes  may  streamline  participants'  experience  as  well  as  provide 
more  rich  data  for  evaluation.  Several  participants  mentioned  their  desire  to  receive  handouts 
corresponding  to  the  presentation  upon  arrival  to  the  course.  One  challenge  in  course  delivery  was 
receiving  final  Power  Point  presentations  from  instructors  in  advance  of  the  course.  In  response,  CIR  has 
assumed  responsibility  for  creating  course  content  and  will  prepare  all  slides  for  presentation  for  future 
iterations  of  the  CE  courses.  This  will  allow  CIR  to  distribute  handouts  at  the  beginning  of  each  course. 

Changes  to  evaluation  procedure  are  recommended  to  acquire  more  rich  feedback  from  participants 
and  to  reduce  participant  fatigue: 

•  Rather  than  have  participants  generate  their  own  ID  code,  assign  each  participant  a  unique 
identifier  at  the  beginning  of  the  CE  series.  This  identifier  will  be  used  throughout  the  CE  series 
and  allow  for  assessment  at  the  individual  level. 

•  Revise  questions  to  indicate  directionality.  For  example,  some  of  the  affect  questions  ask 
participants  to  indicate  their  satisfaction  with  the  depth,  breadth,  and  appropriate  delivery  level 
of  the  course.  In  cases  where  participants  indicate  that  they  disagree,  it  is  not  clear  whether 
their  disagreement  indicates  that  the  courses  went  in  too  much/not  enough  depth,  where  too 
broad  or  too  narrow,  or  delivered  at  a  too  basic  or  too  advanced  level. 

•  When  asked  to  indicate  their  current  practice  setting  (i.e.,  aging,  child  welfare,  health,  mental 
health,  substance  abuse),  most  providers  indicate  at  least  two,  if  not  several,  practice  areas, 
rendering  this  item  not  particularly  informative.  For  evaluation  purposes,  it  might  be  preferable 
to  ask  participants  to  indicate  their  place  of  behavioral  health  employment  and  then  code  their 
responses  during  data  analysis. 

•  Several  participants  have  indicated  being  too  tired  after  two  full  days  of  courses  to  respond  to 
evaluation  measures  with  sufficient  energy.  The  following  adjustments  are  proposed: 

o  Flave  participants  complete  demographic  items  only  once  -  prior  to  the  beginning  of  the 
first  CE  course  they  take  in  the  series.  This  will  eliminate  the  need  for  participants  who 
have  taken  more  than  one  course  to  provide  redundant  information.  For  all  participants, 
this  will  reduce  the  time  needed  to  complete  measures  at  the  end  of  the  two  days  of 
courses. 

o  Instead  of  asking  participants  to  complete  one  long  measure  at  the  end  of  the  course, 
allow  participants  to  provide  open-ended  reactions  on  a  form  throughout  both  days. 
Participants  will  still  provide  feedback  at  the  completion  of  the  course,  but  this  will 
allow  them  to  provide  more  in-depth  feedback  when  they  are  so  compelled,  rather  than 
attempting  to  recall  thoughts  on  specific  content  areas  for  15  hours  of  instruction  at  the 
very  end. 


14 


Appendix  K 

Curriculum  Vitae  for  Dr.  Jeffery  Wilkins 


Distribution  Limited  to  U.S.  Government  Agencies  Only 

CV,  Jeffery  N.  Wilkins,  M.D. 

September,  2010 


Curriculum  Vitae 
JEFFERY  N.  WILKINS,  M.D. 

Mailing  Address:  Jeffery  N.  Wilkins,  M.D.,  D.F.A.P.A. 

LINCY/Heyward-Moynihan  Chair  in  Addiction  Medicine 

Vice  Chair,  Department  of  Psychiatry 

Cedars-Sinai  Medical  Center 

8730  Alden  Drive,  Room  E-130 

Los  Angeles,  CA  90048 


EDUCATION 

1968 

1972 

1972 

1972-1973 

1974-1977 


Bachelor  of  Science  (Cum  Laude),  University  of  Notre  Dame,  South  Bend  IN 
M.D.,  University  of  California  San  Diego  School  of  Medicine,  La  Jolla  CA 
Neurology,  Queen's  Square  and  St.  Pancras  Hospitals,  London,  England 
Medicine  Internship,  Univ.  of  Calif.  San  Diego  School  of  Medicine,  La  Jolla  CA 
Psychiatry  Residency,  UCLA  Neuropsychiatric  Institute,  Los  Angeles  CA 


LICENSURE  AND  DIPLOMATE  STATUS 

State  of  California,  No.  G  25901 

1979  Diplomate,  American  Board  of  Psychiatry  and  Neurology 

1994  Added  Qualification  in  Addiction  Psychiatry  (American  Board  of  Psychiatry  and  Neurology) 

2004  Certification  by  Exam  from  American  Society  of  Addiction  Medicine 

2005  Recertification:  Added  Qualification  in  Addiction  Psychiatry  (American  Board  of  Psychiatry 
and  Neurology) 


HONORS  AND  AWARDS 

1.  March  of  Dimes  Fellowship  in  Medicine:  1969-1970;  1970-1971;  1971-1972 

2.  Pharmaceutical  Manufacturers  Assn.  Foundation  Fellowship  in  Clinical  Pharmacology:  1970- 
1971;  1971-1972 

3.  Veterans  Administration  Career  Development  Award  (Research  Associate):  1978-1981 

4.  Inducted  as  Fellow,  American  Psychiatric  Association  Fellow,  May  2002,  Philadelphia,  PA. 

5.  Inducted  as  Distinguished  Fellow,  American  Psychiatric  Association,  January,  2003 

6.  Inducted  as  Fellow,  American  Society  of  Addiction  Medicine,  2007 

7.  Elected  Member,  Medical  Executive  Committee,  Cedars-Sinai  Medical  Center:  2007-2010 

8.  Member,  Board  of  Directors,  Psychological  Trauma  Center:  2006-Ongoing 

9.  Member,  Board  of  Directors,  Brent  Shapiro  Foundation:  2007-Ongoing 

10.  Member,  Board  of  Directors,  The  Lincy  Foundation:  2008-Ongoing 

11.  President-Elect,  California  Society  of  Addiction  Medicine:  2010 


POSITIONS 

2007-Present 

2006-Present 

2003-Present 

2000-Present 

2000-Present 


Director,  Health  and  Human  Services  Policy,  The  Lincy  Foundation 
LINCY/Heyward-Moynihan  Endowed  Chair  in  Addiction  Medicine 
Vice  Chairman,  Dept,  of  Psychiatry,  Cedars-Sinai  Medical  Center 
Director,  Addiction  Medicine,  Dept,  of  Psychiatry,  Cedars-Sinai  Medical  Center 
Director,  Addiction  Studies,  Clinical  Trials  Unit,  Department  of  Psychiatry,  CSMC 
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2000-  2001  Interim  Director,  Residency  Education,  Dept,  of  Psychiatry,  Cedars-Sinai  Medical  Ctr. 

1994-Present  Adjunct  Professor  of  Psychiatry  and  Biobehavioral  Sciences,  UCLA  School  of  Medicine 

1977- Present  Director,  Clinical  Psychopharmacology  Unit,  Greater  Los  Angeles  VA  Healthcare  Ctr. 

1999-Present  Research  Director,  U.S.  VETS,  a  national  organization  specializing  in  providing 

housing  and  other  services  to  homeless  veterans 
1 994-1 999Medical  Director,  Comprehensive  Homeless  Ctr.  of  Excellence,  GLAVA  Healthcare  Ctr. 
1991-1994Associate  Chief  of  Psychiatry  for  Substance  Abuse  Programs,  WLA  V.A.  Medical  Center 
1 985-1 994Adjunct  Assoc.  Professor  of  Psychiatry  &  Biobehavioral  Sciences,  UCLA  School  of  Medicine 
1 985-1 990Program  Chief,  Treatment  Refractory  Unit,  West  LA  V.A.  Medical  Center 
1981-1985Program  Chief,  Forensic  Unit  (Crisis  Oriented  Psychiatric  Evaluation  and 

Stabilization  Unit,  C.O.P.E.S.),  West  LA  V.A.  Medical  Center 

1 978- 1 985Assistant  Professor,  Dept,  of  Psychiatry,  UCLA  School  of  Medicine 
1978-1981  Research  Associate,  Veterans  Administration  Division  of  Medicine 

Career  Development  Program 

1 977-1 978Program  Chief,  Combined  Alcohol  and  Drug  Inpatient  Treatment,  VAMC,  West  LA 
RESEARCH  INTERESTS 

Resilience  and  Prevention  of  Substance  Abuse  in  Children  and  Adolescents 

Treatment/Pharmacotherapy,  of  adults,  adolescents  and  children  with  substance  abuse  problems  or  substance 
abuse  and  mental  illness 
Pharmacokinetics  of  substances  of  abuse 
Pharmacokinetics  of  psychotropic  medication 

Identification  and  utilization  of  biological  markers  in  the  diagnosis  and  treatment  of  substance  abuse  and 
mental  health  disorders 

ACTIVE  CLINICAL  TRIALS  and  OTHER  GRANT/CONTRACT  ACTIVITIES 


ACTIVE  STUDIES 

Alcoholism,  Genes  and  Hormones.  Tissue  Repository  ProOOO  19076  (Co-Investigator;  Principal  Investigator, 
Magdalena  Uhart.)  STATUS:  Human  Subjects  Actively  Involved  in  Study 

Cocaine  and  Sympathetic  Nerve  Activity  in  Humans  ProOOO  19549  (Co-Principal  Investigator;  Principal 
Investigator,  Ron  Victor,  M.D.)  STATUS:  Human  Subjects  Completed;  Data  under  analysis 

Evaluation  of  Prometa  Treatment  Protocol  for  Treatment  of  Alcohol  Dependence  Pro00006764 
STATUS:  Human  Subjects  Completed;  Data  under  analysis 


PENDING  GRANTS 

1 R0 1 DA02 1249-01 A2  (Pechnick)  12/01/07  -  1 1/30/12  0.60  calendar 

National  Institute  on  Drug  Abuse 

Nicotine  Addiction:  Influence  of  Prenatal  and  Adolescent  Exposure 

The  goal  of  this  study  is  to  understand  the  relationships  between  nicotine-induced  changes  in  behavior  and 
changes  in  brain  function,  by  studying  the  effects  of  prenatal  exposure  to  nicotine,  specifically  the  nicotinic 
cholinergic  receptor,  on  mRNA  expression  in  adulthood. 

Role:  Co-Investigator.  Dr.  Wilkins  will  oversee  the  laboratory  analysis  of  nicotine  and  nicotine  metabolites 
in  body  fluids  from  study  animals 
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(Shoptaw)  7/1/08  -  6/30/12  1.80  calendar 

Regents  of  the  UC  (NIDA) 

Antipsychotic  Treatment  of  Methamphetamine-Induced  Psychosis 

The  major  goal  of  this  project  is  to  examine  the  efficacy  of  using  ongoing  atypical  antipsychotic  medication 
(aripiprazole)  compared  to  placebo  in  the  presence  of  cognitive  behavioral  drug  abuse  counseling  in 
individuals  with  methamphetamine-induced  psychosis  after  acute  stabilization. 

Role:  Site-  Principal  Investigator  Dr.  Wilkins  will  direct  the  evaluation  and  treatment  of  methamphetamine- 
induced  psychosis  in  the  study  participants  of  this  study,  as  well  as  oversee  the  collection  and  analysis  of 
laboratory  and  clinical  assessment  inventories  for  the  study. 

ACTIVE  GRANTS 

CSR200892  (Wilkins)  '  8/01/05-02/29/09  0.60  calendar 

Hythiam,  Inc. 

Evaluation  of  HANDS  (PROMETA)  Protocol  for  Treatment  of  Alcohol  Dependence 

The  purpose  of  this  study  is  to  evaluate  the  clinical  efficacy  of  Prometa,  a  medication-based  intervention  for 

treatment  of  alcohol  dependence,  and  to  decrease  alcohol  use  and  increase  abstinence  from  alcohol. 

Role:  Principal  Investigator 

PRIOR  GRANTS 

Principal  Investigator  and  co-investigator  on  10  clinical  trials  and  10  additional  grants/contracts,  at  least  two 
of  which  have  resulted  in  FDA  filing  of  two  New  Drug  Applications  (NDA).  Activities  have  included 
protocol  development,  RFA  development,  subject  recruitment,  medical  oversight,  and  supervisor  of  core 
laboratory  for  dependent  and  independent  laboratory  variables.  NDA’s  were  for  LAAM  (1-alpha-acetyl 
methadol),  a  long  acting  opioid  agonist  used  for  opioid  maintenance,  and  buprenorphine  and  buprenorphine  + 
naloxone,  shorter  acting  opioid  maintenance  medications. 

CLINICAL  TRIALS  GRANTS  and  CONTRACTS 

1.  PRINCIPAL  INVESTIGATOR,  Sponsor:  Alkermes.  Project:  Vivitrol  Naltrexone  in  combination  with 
psychosocial  treatment  for  treatment  of  alcohol  dependent  adults.  June,  2002-July,  2007 

2.  CO-INVESTIGATOR  (10%)  on  NIDA  Contract  from  the  Medication  Development  Division  "DHEA 
TREATMENT  OF  COCAINE  DEPENDENCE”  (Walter  Ling,  M.D.,  Principal  Investigator,  Steven 
Shoptaw,  Ph.D.,  Co-Principal  Investigator).  Duration  10/1/98-9/30/00.  The  goal  of  this  protocol  is  to 
assess  the  ability  of  orally  administered  DHEA  to  alter  the  relapse  rate  and  study  retention  of  patients  with 
cocaine  dependence.  Dr.  Wilkins  laboratory  is  responsible  for  laboratory  analysis  of  circulating  DHEA, 
DHEAS,  cortisol,  CRF  levels  as  well  as  quantitative  urine  substance  abuse  testing. 

3.  CO-INVESTIGATOR  (10%)  on  NIDA  Contract  from  the  Medication  Development  Division  "RAPID 
ASSESSMENT  OF  PHARMACOTHERAPY  OF  COCAINE  DEPENDENCE"  (Walter  Ling,  M.D., 
Principal  Investigator).  Duration  10/1/96-9/30/99.  The  goal  of  this  protocol  is  to  assess  new  potential 
pharmacotherapeutic  agents  for  the  treatment  of  cocaine  abuse/dependence.  The  first  medication  studied 
was  amantadine.  Results  from  this  study  were  positive  for  reduction  of  cocaine  abuse  and  increased  stays 
in  treatment. 

4.  CO-INVESTIGATOR  (10%)  on  NIDA  Contract  from  the  Medication  Development  Division  "KINETICS 
OF  LAAM  FOR  TREATMENT  OF  OPIOID  DEPENDENCE”  (Walter  Ling,  M.D.,  Principal 
Investigator).  Duration  10/1/96-9/30/99.  The  goal  of  this  protocol  is  to  assess  LAAM  kinetics  in  opioid- 
dependent  individuals  who  receive  LAAM  in  the  context  of  a  psychosocial  treatment  program.  Urine 
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determinations  are  also  performed  for  quantitative  determinations  of  morphine  and  other  substances  of 
abuse  as  well  as  methadone  kinetics  during  washout  transition  to  LAAM  administration. 

5.  PRINCIPAL-INVESTIGATOR  (25%  time,  no  salary)  of  NIDA  grant  #1  R01  DA0655 1  "DESIPRAMINE 
TREATMENT  OF  SCHIZOPHRENIC  COCAINE  ADDICTS’'  funded  04/01/89  through  08/31/94  with 
annual  direct  costs  of . 

6.  PRINCIPAL  INVESTIGATOR  (25%  time,  no  salary)  of  NIDA  grant  #1  R01  DA05685-01, 
"BROMOCRIPTINE  IN  THE  TREATMENT  OF  COCAINE  ABUSE,":  funded  01/01/89  through 
06/30/93 

7.  Director  of  VAMC  WLA  Clinical  Psychopharmacology  Unit  performing  urine  toxicological  analysis  for 
all  sites  NIDA  multicenter  trial  of  SELEGILINE  IN  THE  TREATMENT  OF  COCAINE  DEPENDENCE 
(centers  are  VAMC  Washington  DC,  VAMC  Philadelphia,  Univ.  of  Calif.,  San  Francisco,  and  VAMC 
West  Los  Angeles) 

8.  SITE  PRINCIPAL  INVESTIGATOR  and  Director  of  VAMC  WLA  Clinical  Psychopharmacology  Unit 
performing  urine  toxicological  analysis  for  West  LA  site  for  NIDA  multicenter  trial  of  BUPROPRION  IN 
THE  TREATMENT  OF  COCAINE  DEPENDENCE  (centers  were  Yale  University,  VAMC,  VAMC 
Philadelphia,  and  VAMC  West  Los  Angeles). 

9.  Director  of  VAMC  WLA  Clinical  Psychopharmacology  Unit  performing  urine  toxicological  analysis  for 
the  Los  Angeles  Treatment  Research  Unit  site  as  part  of  a  NIDA  multicenter  trial  of  BUPRENORPHINE 
IN  THE  TREATMENT  OF  OPIOID  DEPENDENCE 

10.  Director  of  VAMC  WLA  Clinical  Psychopharmacology  Unit  performing  urine  toxicological  analysis  for 
the  Los  Angeles  Treatment  Research  Unit  site  as  part  of  a  NIDA  multicenter  trial  of  BUPRENORPHINE 
PLUS  NALOXONE  IN  THE  TREATMENT  OF  OPIOID  DEPENDENCE 

OTHER  GRANTS  and  CONTRACTS 

11.  PRINCIPAL  INVESTIGATOR  (10%  time,  no  salary)  of  "NEUROPSYCHOLOGICAL  AND 
BIOMEDICAL  ASSESSMENT  OF  COCAINE  ABUSERS"  (W.  van  Gorp,  Co-Principal  Investigator). 
VA  Merit  Review  funded  10/01/92  to  09/30/97 

.  The  study  assessed  neuropsychological  function  in  100 

cocaine  abusing  patients  without  alcohol  dependence  with  corresponding  100  "substance-free"  controls 
assessed  at  the  same  intervals  of  72  hours  and  21  days  following  inpatient  admission,  6,  12,  18,  and  24 
months  post  discharge.  An  additional  nested  control  group  of  30  subjects  are  being  assessed  to  determine 
the  impact  of  repeated  test  administration.  All  subjects,  including  controls,  were  screened  throughout  the 
study  for  the  presence  of  cocaine  metabolite  and  other  substances  of  abuse.  The  neuropsychological 
results  are  being  compared  to  circulating  prolactin  and  additional  neuroendocrines. 

12.  CO-INVESTIGATOR  (5%  time,  no  salary)  of  NIDA  grant  ROl  DA  09436  "SKILLS  TRAINING  FOR 
SUBSTANCE  ABUSING  SCHIZOPHRENICS"  (A.  Shaner,  M.D.,  Principal  Investigator),  a  NIDA  Phase 
I  development  project  funded  9/30/94  through  8/31/97 

.  The  protocol  is  structured  as  a  prospective,  two-year  longitudinal  follow-up  of  6 
cohorts  of  5  subjects.  The  patients  received  specialized  cognitive  skills  training  through  the  UCLA 
Schizophrenia  Clinical  Research  Center  (R.  Liberman,  M.D.,  Principal  Investigator)  following  the  relapse 
prevention  model  of  Marlatt  et  al.  In  the  role  as  Director  of  the  Clinical  Psychopharmacology  Unit,  Dr. 
Wilkins  was  responsible  for  laboratory  determination  of  parent  compounds  and  metabolites  of  cocaine, 
methamphetamine,  and  other  substances  of  abuse.  The  substance  abuse  measures  will  serve  as  primary 
outcome  variables  for  the  study. 

13.  CO-INVESTIGATOR  (5%)  in  MRI  Studies  in  VA  Merit  Review  "BIPOLAR  DISORDER  AND 
ALCOHOL  DEPENDENCE"  (Lori  Altshuler,  M.D.,  Principal  Investigator).  This  protocol  had  two 
primary  goals:  1)  to  perform  MRI  scans  on  bipolar  patients  with  particular  focus  on  the  temporal  lobe  and 
hippocampus,  and  2)  to  follow  bipolar  patients  with  and  without  alcohol  dependence  to  assess  if  the 
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comorbid  disorder  is  predictive  of  a  worse  prognosis,  that  is,  whether  alcohol  comorbidity  is  a 
vulnerability  marker  for  a  worse  course  outcome  (measures  include  affective  instability,  dropout  from 
treatment  or  study,  rates  of  rehospitalization  and  social/occupational  function). 

14.  CO-INVESTIGATOR  (10%  time,  no  salary)  of  "TREATMENT  OF  SCHIZOPHRENIA  AND 
SUBSTANCE  ABUSE"  (A.  Shaner,  M.D.,  Principal  Investigator),  funded  4/01/91  to  08/31/95  with  annual 
total  costs  of 

15.  CO-INVESTIGATOR  (10%  time,  no  salary)  on  NIMH  protocol  "Treatment  of  the  Stimulant-Abusing 
Schizophrenic"  (MH48081,  A.  Shaner,  M.D.,  Principal  Investigator),  approved  for  funding  10/01/90  to 
9/30/94,  . 

16.  CO-INVESTIGATOR  (10%  time,  no  salary)  of  "TREATMENT  OF  SCHIZOPHRENIA  AND 
SUBSTANCE  ABUSE"  (A.  Shaner,  M.D.,  Principal  Investigator),  HSR&D  VA  Research  Grant  (HR  #90- 
03)  funded  04/01/91  to  04/01/95 

17.  CO-INVESTIGATOR  (10%  time,  no  salary)  on  NIDA  application  "CARDIOVASCULAR  EFFECTS 
AND  TOXICITY'S  OF  COCAINE  (long-term)  (K.  Nademanee,  M.D.,  Principal  Investigator),  approved 
for  funding  4/01/90  to  3/31/92,. 

18.  CO-INVESTIGATOR  (2.5%  time,  no  salary)  on  NIDA  application  "NEUROENDOCRINE  EFFECTS  OF 
PHENCYCLIDINE  IN  THE  RAT"  (R.  George,  Ph.D.,  Principal  Investigator)  approved  for  funding  4/1/90 
to  3/31/93. 

19.  CO-INVESTIGATOR  (5%)  Norman  Cousins  Foundation  for  Immunological  Research  on 
IMMUNOLOGICAL  FUNCTION  IN  CAREGIVERS  (Stacy  Wilkins,  Ph.D.,  Principal  Investigator). 
Duration  10/1/97-9/30/99.  The  study  provides  a  comparison  of  immune  function  in  depressed  spouse 
caregivers  of  demented  spouses  and  sex  and  age  matched  elderly  controls.  Dr.  Wilkins’  laboratory 
performs  assessments  of  the  hypothalamic-pituitary-adrenal  axis  for  the  protocol.  Both  caregivers  and 
controls  are  subjected  to  brief  human  laboratory  stressors  with  serial  assessment  of  HP  A  and  immune 
function. 

20.  CO-INVESTIGATOR  (5%)  on  NIDA  application  "CONTINGENCY  MANAGEMENT-  TOBACCO 
SMOKING  IN  OPIATE  ADDICTS"  (Steven.  Shoptaw,  Ph.D.,  Principal  Investigator). ).  Duration 
10/1/96-9/30/99. 

MEMBERSHIP:  SCIENTIFIC  ADVISORY  BOARDS 

Alkermes,  Inc.  (2005  -  2009) 

Cephalon,  Inc.  (2005  -  2009) 

DIC  Entertainment  (2004  -  2009) 

Hythiam,  Inc.  (2004  -  2008) 

COMMITTEES 

2004-Present 
2002-Present 
2002-Present 
2002-Present 
1999-2001 

1992- 1997 

1993- 1994 
1987-1989 

1986-1989 


SAE  Review  Committee,  Cedars-Sinai  Medical  Center 
HIPAA  Advisory  Task  Force,  Cedars-Sinai  Medical  Center 
Well  Being  Commitee,  Cedars-Sinai  Medical  Center 
IRB,  Cedars-Sinai  Medical  Center 

Member,  California  Dept,  of  Mental  Health  AB  34  Advisory  Committee 
VAMC  WLA  Research  and  Development  Committee, 

UCLA  School  of  Medicine  Dean's  Committee  on  Neuroscience  Education 
Human  Subject  Protection  Committee,  W.  Los  Angeles  V.A.  Medical 
Center,  Brentwood  Division 

Pharmacy  and  Therapeutics  Committee,  W.  Los  Angeles  V.A.  Medical  Center 
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1983- 1985 

1984- 1993 

1981-1982 

1981-1985 

1980-1993 

1979-1982 

1971-1972 


Academic  Appointments  and  Advancements  Committee,  U.C.L.A.  Division 
of  Adult  Psychiatry /Biobehavioral  Sciences 
Consulting  Committee,  Psychopharmacology  Course,  UCLA/VA, 

Brentwood  Psychiatry  Residency 

Academic  Appointments  and  Advancements  Committee,  U.C.L.A.  Division 
of  Adult  Psychiatry /Biobehavioral  Sciences  (alternate) 

Residency  Education,  Inpatient  Psychiatry 

Research  Safety  Subcommittee,  Los  Angeles  VA  Medical  Center,  Brentwood  Divn. 
Research  and  Development  Committee,  Los  Angeles  VA  Medical  Center- 
Brentwood  Division 

Admissions  Committee,  School  of  Medicine,  University  of  California, 

San  Diego 


EXPERIENCE  IN  ADMINISTRATION 

Administration  of  VA  Substance  Abuse  Programs  ( 1991-1994) 

a)  Passage  of  JCAHO  review  of  Substance  Abuse  Programs  in  August,  1993 

b)  Establishment  of  Psychiatry  Residency  and  UCLA  Medical  Student  Teaching  Program  within 

VAMC  WLA  Substance  Abuse  Programs 

c)  Establishment  of  Staff  Training  Program  leading  to  State  of  California  Certificate  in  Addiction 

Counseling 

d)  Supervision  of  VAMC  WLA  Medication  Development  Unit  clinical  research  staff 

e)  Introduction  of  computerized  treatment  planning  system  in  Substance  Abuse  Programs 

(example  database  of  352  consecutive  patients) 

f)  Introduction  of  computerized  patient  tracking  system  for  the  VAMC  WLA  Homeless  Center 

Administratration  of  Analytical  Laboratory  for  National  Institute  on  Drug  ttAbuse  Multicenter 
Trials 

a)  Director  of  VAMC  WLA  Clinical  Psychopharmacology  Unit  performing  urine  toxicological 
analysis  for  all  sites  NIDA  multicenter  trial  of  selegiline  in  the  treatment  of  cocaine 
dependence  (centers  are  VAMC  Washington  DC,  VAMC  Philadelphia,  Univ.  of  Calif.,  San 
Francisco,  and  VAMC  West  Los  Angeles) 

b)  Director  of  VAMC  WLA  Clinical  Psychopharmacology  Unit  performing  urine  toxicological 
analysis  for  West  LA  site  for  NIDA  multicenter  trial  of  buproprion  in  the  treatment  of  cocaine 
dependence  (centers  were  Yale  University,  VAMC,  VAMC  Philadelphia,  and  VAMC  West 
Los  Angeles) 

c)  Director  of  VAMC  WLA  Clinical  Psychopharmacology  Unit  performing  urine  toxicological 
analysis  for  the  Los  Angeles  Treatment  Research  Unit  site  as  part  of  a  NIDA  multicenter  trial 
of  buprenorphine  in  the  treatment  of  opioid  dependence 

Provision  of  laboratory  services 

In  parallel  with  the  VA’s  goal  to  increase  service  while  decreasing  costs,  the  CPU  Laboratory 
targeted  increasing  analytical  efficiency  as  well  as  increasing  available  methods.  For  example, 
as  clinical  and  research  programs  have  had  to  exist  on  reduced  budgets,  the  CPU  had  to  meet 
market-driven  reductions  in  per  sample  costs  combined  with  demand  for  development  of  new 
assays.  Cost  reductions  were  produced  through  changes  in  procedures  of  the  analytical  process, 
including  the  development  of  new  modes  of  sample  extraction,  employment  of  automatic 
pipetting  systems  and  improved  detection  systems. 
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The  ability  of  the  CPU  to  cut  costs  allowed  it  to  provide  urine  analysis  for  the  Comprehensive 
Homeless  Center  of  Excellence  Programs.  Substance  abuse  urine  results  as  well  as  other 
outcome  measures  taken  by  our  group  were  cited  in  HR3031,  a  House  of  Representatives  Bill 
supporting  the  use  of  VA  funds  for  large-scale  residential  programs  for  homeless  veterans. 

Teaching  based  administrative  contributions 

a)  Introducing  residents  and  medical  students  to  prevention  and  treatment  of  aggression  and 
violence.  Extension  of  this  training  to  nurses  and  social  workers. 

b)  Establishment  of  residence  training  in  substance  abuse  at  the  VA  for  UCLA-based  residents  as 
well  as  Martin-Luther  King  psychiatry  residents 

c)  Establishing  administrative  links  and  financial  support  that  allowed  VA  counselors  to  enter 
training  at  Cal  State  Dominguez  Hills  where  they  were  able  to  receive  licenses  in  State  of 
California  Accreditation  in  Drug  and  Alcohol  Counseling 

FOREIGN  LANGUAGE 

Use  of  Written  and  Spoken  Spanish  in  the  Evaluation  and  Treatment  of  Patients 
TEACHING:  Information  Regarding  Academic  and  Community  Teaching  Available  on  Request 

REVIEWER 

Experimental  Clinical  Psychopharmacology  (2006) 

Pharmacology,  Biochemistry  and  Behavior  (2006) 

Schizophrenia  Bulletin  (1999) 

Journal  of  Neuropsycho  immunology  (1999) 

Archives  of  General  Psychiatry  (1987,  1989,  1990,  1994) 

Psychoneuroendocrinology  (1988) 

Journal  of  Clinical  Pharmacology  and  Therapeutics  (1988) 

American  Journal  of  Public  Health  (1988,  1989,  1994,  1996) 

Hospital  and  Community  Psychiatry  (1988,  1989) 

Psychopharmacology  (1990) 

VA  STUDY  SECTION  REVIEW 

Health  Services,  Research  and  Development  Service,  Department  of  Veterans  Affairs,  Veterans  Health 
Administration:  October  96;  April  97,  October  97,  April  98,  October  98,  April  99,  October  99,  April 
2000 

VA  Merit  Review:  September,  2000. 

NIDA  STUDY  SECTION  REVIEW 

March,  1999:  Center  review 

March  1997,  October  1997:  Pharmacotherapy  of  Stimulant  Dependence 

TEACHING 

a)  UCLA  School  of  Medicine  Neuroscience  Education  Committee  (Chaired  by  Dept,  of 
Neurology  and  Pharmacology,  1993-94) 

b)  Prevention  and  Treatment  of  Aggression  and  Violence  (UCLA  Medical  Students,  VA 
Nursing,  VA  Social  Work,  1981-1994) 
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c)  Diagnosis  and  Treatment  of  Substance  Abuse  (Psychiatry  Residents,  Medical  Students,  VA 
Nursing,  VA  Social  Work,  1981-1994) 

d)  UCLA  98F  -  Substance  abuse  in  the  90’s  (Undergraduate  honor’s  course,  1994,  1995) 

e)  UCLA  Individual  laboratory  projects  (Psychiatry  199,  1981-95) 

f)  UCLA  Student  Research  Projects  (SRP  program,  1985-1995) 

g)  UCLA  Medical  Students  (Series  on  Substance  Abuse  in  General  Psychiatry  Series,  B.  Guze, 

M.D.,  Coordinator,  1991-1998) 

h)  Substance  Abuse  for  Psychologists  (Steven  Sideroff,  Ph.D.,  Coordinator,  1991 -Ongoing) 

i)  Training  seminars  and  supervision  to  Social  work  interns,  Americorps  volunteers  in  the 
homeless  programs,  and  provided  speakers  and  educational  programs  to  the  staff  of  the 
VA’s  Comprehensive  Homeless  Center  (1994-2000) 

j)  Development  of  UCLA  Medical  Student  Curriculum  for  psychiatry  at  Cedars-Sinai  Medical 
Center  (2000-  ongoing) 

k)  Course  Chair,  UCLA  School  of  Medicine  Course  PS250.04,  Advanced  Clinical  Clerkship  in 
Psychiatry  (2000-  ongoing) 

l)  Course  Coordinator,  “Biological  Perspective  on  Dual  Diagnosis”  (UCLA  Extension  10- 
week,  30  hour  course  for  Los  Angeles  County  Department  of  Mental  Health  and  Division  of 
Drug  and  Alcohol  Counselors)  (2001-  ongoing) 

m)  Coordinator,  Training  in  Substance  Abuse  for  Psychiatry  Residents  and  UCLA  Medical 

Students  at  Cedars-Sinai  Medical  Center  (2000-  ongoing) 

In  addition  to  the  above  activities,  our  group  has  supported  career  advancement  for  12  staff  into 
professional  fields.  Coming  to  the  CPU  staff  from  UCLA,  the  California  State  Colleges,  and 
USC,  we  have  had  staff  ultimately  leave  the  CPU  to  enter  the  following  graduate  training: 

Medical  School:  UCLA  (1),  UCSD  (1),  Stanford  (1),  USC  (1) 

Medical  College  of  Pennsylvania  (1),  S.U.N.Y.  (1), 

Doctoral  Programs:  Pharmacology,  UCLA  (1),  Neuroscience,  Berkeley  (1), 

Clinical  Psychology,  UCLA  (3),  Clinical  Psychology,  UCSF  (1) 

PUBLICATIONS 

1.  Sinha,  Y.N.,  Wilkins,  J.N.,  Selby,  F.  and  VanderLaan,  W.P.  Pituitary  and  serum  growth  hormone 

during  under-nutrition  and  catch-up  growth  in  young  rats.  Endocrinology  92(6):  1768- 1771, 

1973. 

2.  Wilkins,  J.N.,  Mayer,  S.E.  and  VanderLaan,  W.P.  The  effects  of  hyperthyroidism  and  2,4- 

dinitrophenol  on  growth  hormone  synthesis.  Endocrinology  95(5):  1259-1267,  1974. 

3.  Wilkins,  J.N.  Humors,  Humans  and  Behavior.  Contemporary  Psychology  22(1 1):8 1 9-82 1 ,  1977. 

4.  Jarvik,  L.F.  and  Wilkins,  J.N.  Aging,  hormones  and  mental  function.  In: 

Neuropsychopharmacology.  P.  Deniker,  C.  Radoco-Thomas  and  A.  Villaneuve  (eds.),  New 
York,  Pergamon  Press,  Ltd.,  Oxford,  pp.  49-57,  1978. 

5.  Wilkins,  J.N.  Endocrines  and  Depression.  In:  Contemporary  Models  in  Consultation-Liaison 

Psychiatry.  R.O.  Pasnau,  et  al.  (eds.),  New  York,  Spectrum  Publications,  pp  173-184,  1978. 
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6.  Wilkins,  J.N.  Neuroendocrinology  and  Clinical  Psychiatry.  In:  Psychiatric  Research,  in  Practice, 

E.A.  Serafetinides  (ed.),  M.  Greenblatt  (Series  ed.),  New  York.  Grune  and  Stratton,  pp.  101- 
111,1981. 

7.  Van  Putten,  T.,  May,  P.R.A.  and  Wilkins,  J.N.  Importance  of  akinesia:  plasma  chlorpromazine 

and  prolactin  levels.  American  Journal  of  Psychiatry  137(11):  1446-1448,  1980. 

8.  Van  Putten,  T.,  May,  P.R.A.,  Marder,  S.R.  and  Wilkins,  J.N.  Plasma  levels  of  thiothixene  by 

radioreceptor  assay  and  clinical  outcome.  Psychopharmacology  Bulletin  1 8(1):99- 101,  1982. 

9.  Cohen,  L.S.,  Gosenfeld,  L.,  Wilkins,  J.N.,  Kammerer,  R.C.,  and  Tachiki,  K.  Demonstration  of  an 

amino  acid  metabolite  of  phencyclidine  (Letter  to  the  Editor).  New  England  Journal  of 
Medicine,  306(23):  1427-1428,  1982. 

10.  Wilkins,  J.N.,  Carlson,  H.E.,  Van  Vunakis,  H.,  Hill,  M.A.,  Gritz,  E.  and  Jarvik,  M.E.  Nicotine 
from  cigarette  smoking  increases  circulating  levels  of  cortisol,  growth  hormone  and  prolactin  in 
male  chronic  smokers.  Psychopharmacology  78(4):305-308,  1982. 

11.  Gerner,  R.H.  and  Wilkins,  J.N.  CSF  Cortisol  in  patients  with  depression,  mania  or  anorexia 
nervosa  and  in  normal  subjects.  American  Journal  of  Psychiatry  140(l):92-94,  1983. 

12.  Carlson,  H.E.,  Wasser,  H.L.,  Levin,  S.R.,  and  Wilkins,  J.N.  Prolactin  stimulation  by  meals  is 
related  to  protein  content.  Journal  of  Clinical  Endocrinology  and  Metabolism,  57(2):334-338, 
1983. 

13.  Marder,  S.R.,  Swann,  E.,  Winslade,  W.J.,  Van  Putten,  T.,  Chien,  C.P.  and  Wilkins,  J.N.  A  study 
of  medication  refusal  by  involuntary  psychiatric  patients.  Hospital  and  Community  Psychiatry, 
35(7):724-726,  1984. 

14.  Kook,  K.A.,  Stimmel,  G.L.,  Wilkins,  J.N.,  and  Sprangher,  G.G.  Accuracy  and  safety  of  a  priori 
lithium  loading.  J  of  Clin  Psychiatry,  46(2):49-51,  1985. 

15.  Escobar,  J.I.,  Mann,  J.J.,  Keller,  J.,  Wilkins,  J.N.,  Mason,  B.  and  Mills,  M.J.  Comparison  of 
Injectable  Molindone  and  Haloperidol  Followed  by  Oral  Dosage  Forms  in  Acutely  Ill 
Schizophrenics.  Journal,  of  Clinical  Psychiatry,  46(8):  15-19,  1985. 

16.  Gerner,  R.H. ,  and  Wilkins,  J.N.  CSF  cortisol  in  affective  illness,  Psychiatric  Medicine,  3(1): 
33-40,  1985. 

17.  Gorelick,  D.A.,  and  Wilkins,  J.N.  Special  aspects  of  human  alcohol  withdrawal.  In:  Recent 
Developments  in  Alcoholism,  Volume  IV,  Ch.  13,  pp.  283-305,  (ed.,  M.  Galanter),  Plenum 
Publishing  Corp.,  1986. 

18.  Wilkins,  J.N.,  and  Gorelick,  D.A.  Clinical  Neuroendocrinology  and  Neuropharmacology  of 
Alcohol  Withdrawal.  In:  Recent  Developments  in  Alcoholism,  Volume  IV,  Ch.  11,  pp  24 1  - 
263,  (ed.,  M.  Galanter),  Plenum  Publisher  Corp.,  1986. 
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19.  Gorelick,  D.A.,  Wilkins,  J.N.,  and  Wong,  C.  Diagnosis  and  treatment  of  chronic  phencyclidine 
(PCP)  abuse.  Phencyclidine:  An  Update,  NIDA  Research  Monograph  64  (ed.,  D.  H.  Clouet), 
pp  218-228,  1986. 

20.  Wilkins,  J.N.  Hallucinogens:  Neurochemical,  Behavioral,  and  Clinical  Perspectives.  The 
Quarterly  Review  of  Biology,  61(1):  146,  1986. 

21.  Baxter,  L.R.,  Wilkins,  J.N.,  and  Smith,  G.B.  A  possible  diurnal  variation  in  trazodone 
metabolism.  Journal  of  Clinical  Psychopharmacology  6(4):223-226,  1986. 

22.  Baxter,  L.R.  Jr.,  Liston,  E.H.,  Schwartz,  J.M.,  Altshuler,  L.L.,  Wilkins,  J.N.,  Richeimer,  S.,  and 
Guze,  B.H.  Prolongation  of  the  antidepressant  response  to  partial  sleep  deprivation  by  lithium. 
Psychiatry  Research  19:17-23,  1986. 

23.  Altshuler,  L.L.,  Kagan,  B.L.,  Baxter,  L.R.  Jr.,  Smith,  G.B.,  and  Wilkins,  J.N.  Effect  of 
interrupted  sleep  patterns  and  partial  sleep  deprivation  on  DST  and  mood  in  psychiatric  house 
officers.  ACTA  Psychiatric  Scandinavica,  75:614-618,  1987 

24.  Wilkins,  J.N.,  Marder,  S.R.,  Van  Putten,  T.,  Midha,  K.K.,  Mintz,  J.,  Setoda,  D.,  and  May,  P.R.A. 
Circulating  prolactin  predicts  risk  of  exacerbation  in  patients  on  depot  fluphenazine. 
Psychopharm  Bulletin  23(3):522-525,  1987. 

25.  Haracz,  J.L.,  Minor,  T.,  Wilkins,  J.N.  and  Zimermann,  E.G.  Learned  helplessness:  An 
experimental  model  of  the  DST  in  rats.  Biological  Psychiatry  23:388-396,1988. 

26.  Haracz,  J.L.,  Minor,  T.,  Wilkins,  J.N.  and  Zimermann,  E.G.  Animal  models:  Promise  and 
pitfalls  -  Response.  Biol  Psychiatry  26:432-433,  1989  (Letter) 

27.  Gorelick,  D.A.,  and  Wilkins,  J.N.  Inpatient  treatment  of  phencyclidine  (PCP)  abusers  and  users. 
American  Journal  of  Drug  and  Alcohol  Abuse  15(1):  1-12,  1989. 

28.  Hartman,  N.,  Jarvik,  M.E.,  and  Wilkins,  J.N.  Reduction  of  Cigarette  smoking  by  use  of  a 
nicotine  patch.  (Letter  to  the  Editor)  Arch  Gen  Psych  46:289,  1989. 

29.  Wilkins,  J.N.,  Spar,  J.E.,  and  Carlson,  H.E.  Desipramine  increases  circulating  growth  hormone 
in  elderly  depressed  patients:  a  pilot  study.  Psychoneuroendocrinology  14(3):  195-202,  1989. 

30.  Nademanee,  K.,  Gorelick,  D.A.,  Josephson,  M.A.,  Ryan,  M.A.,  Wilkins,  J.N.,  Robertson,  H.A., 
Vaghaiwalle  Mody,  F.,  and  Intarachot,  V.  Mycocardial  ischemia  during  cocaine  withdrawal. 
Annals  of  Internal  Medicine  1 1 1(1 1):  876-880,  1989. 

3 1 .  Gorelick,  D.A.,  Wilkins,  J.N.,  and  Wong,  C.  Outpatient  Treatment  of  PCP  Abusers.  Am  J  Drug 
Alcohol  Abuse  15(4):367-374,  1989. 

32.  Wilkins,  J.N.,  and  Gorelick,  D.A.  Clinical  implications  of  PCP,  NMDA,  and  sigma  opiate 
receptor  research,  NIDA  Monograph  "Committee  on  the  Problems  of  Drug  Dependence  (ed., 
D.H.  Clouet)  1989. 
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33.  Wilkins,  J.N.,  Shaner,  A.L.,  Patterson,  C.M.,  Setoda,  D.,  and  Gorelick,  D.  Discrepancies 
between  patient  report,  clinical  assessment,  and  urine  analysis  in  psychiatric  patients  during 
inpatient  admission.  Psychopharmacology  Bulletin  27  (2):  149-154,  1991. 

34.  Parry,  B.L.,  Gerner,  R.H.,  Wilkins,  J.N.,  Halaris,  A.E.,  Carlson,  H.E.,  Hershman,  J.M.,  Linnoila, 
M.,  Merrill,  J.,  Gold,  P.W.,  Gracely,  R.,  Aloi,  J.,  and  Newton,  R.  CSF  and  endocrine  studies  of 
premenstrual  syndrome.  Neuropsychopharmacology  5(2):  127-137,  1991. 

35.  Lipton,  J.,  Zeigler,  S.,  Wilkins,  J.,  Ellison,  G.  A  silicone  pellet  for  continuous  cocaine: 
comparison  with  continuous  amphetamine.  Pharmacology,  Biochemistry  and  Behavior 
38(4):927-930,  1991. 

36.  Wilkins,  J.N.  Brain,  Lung,  and  Cardiovascular  Interactions  with  Cocaine  and  Cocaine-Induced 
Catecholamine  Effects  in  Cocaine.  In:  Physiological  and  Physiopathological  Effects,  pp  9-19, 
(ed.,  Stimmel,  B.;  guest  eds.  Paredes,  A,  Gorelick,  D),  Haworth  Press  Inc.,  New  York,  1992. 

37.  Wilkins,  J.N.,  Gorelick,  D.A.,  Nademanee,  K.,  Taylor,  A.,  and  Herzberg,  D.S.  Hypothalamic- 
Pituitary  Function  During  Alcohol  Exposure  and  Withdrawal  and  Cocaine  Exposure.  In: 

Recent  Developments  in  Alcoholism,  M.  Galanter  (ed).  Plenum  Press:  New  York.,  Ch.  3,  pp  57- 
71, 1992. 

38.  Wilkins,  J.N.  Brain,  Lung,  and  Cardiovascular  Interactions  with  Cocaine  and  Cocaine-Induced 
Catecholamine  Effects.  Journal  of  Addictive  Diseases  1 1  (4):9- 1 9,  1992. 

39.  Tretjak,  Z.,  Knight,  J.A.,  Wilkins,  J.N.,  Setoda,  D.,  and  Voorhees,  R.  Lipoperoxides  in  sebum  of 
substance  users  and  controls.  Annals  of  Clinical  and  Laboratory  Science  22(4):214-220,  1992. 

40.  Shaner,  A.,  Klialsa,  H.,  Roberts,  L.,  Wilkins,  J.,  Anglin,  D.,  and  Hsieh,  S.C.  Unrecognized 
cocaine  use  among  schizophrenic  patients.  Am  J  Psychiatry  150(5):758-762,  1993. 

41.  Berry,  J.,  Van  Gorp  W.G.,  Herzberg,  D.S.,  Hinkin,  C.,  Boone,  K.,  Steinman,  L.,  and  Wilkins, 

J.N.  Neuropsychological  Deficits  in  Abstinent  Cocaine  Abusers:  Preliminary  findings  after  two 
weeks  of  abstinence.  Journal  of  Drug  and  Alcohol  Dependence  32:  231-237,  1993. 

42.  Wilkins,  JN,  and  DA  Gorelick.  Management  of  phencyclidine,  hallucinogen  and  marijuana 
intoxication  and  withdrawal.  Principles  of  Addiction  Medicine  (American  Society  of  Addiction 
Medicine,  N.  Miller,  Editor)  Section  8,  Chapter  6,  1994. 

43.  Wilkins,  JN,  and  DA  Gorelick.  Management  of  stimulant  intoxication  and  withdrawal.  Principles 
of  Addiction  Medicine  (American  Society  of  Addiction  Medicine,  N.  Miller,  Editor)  Section  11, 
Chapter  6,  1994. 

44.  Wilkins,  JN,  and  DA  Gorelick.  Pharmacologic  therapies  for  other  drugs  and  multiple  drug 
addiction.  Principles  of  Addiction  Medicine  (American  Society  of  Addiction  Medicine,  N. 
Miller,  Editor)  Section  12,  Chapter  6,  1994. 
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45.  Haberny,  KA,  Walsh,  SL,  Ginn,  DH,  Wilkins,  JN,  Garner,  JE,  Setoda,  D,  Bigelow,  GE,  Absence 
of  cocaine  interactions  with  the  MAO-B  inhibitor  selegiline.  Journal  of  Drug  and  Alcohol 
Dependence  39;55-62,  1995. 

46.  Shaner,  A,  Mintz,  J,  Eckman,  T,  Roberts,  L,  Wilkins,  J,  Tucker,  D,  Tsuang,  J.  Disability  income, 
cocaine  use  and  repeated  hospitalizations  among  schizophrenic  cocaine  abusers:  a  government 
sponsored  revolving  door?.  New  England  Journal  of  Medicine  333:777-783,  1995. 

47.  Margolin,  A.;  Kosten,  T.R.;  Avants,  S.K.;  Wilkins,  J.N.;  Ling,  W.;  Beckson,  M.;  Arndt,  L; 
Cornish,  J.;  Li,  S.H.;  Bridge,  P.;  Ascher,  J.A.  Bupropion  for  cocaine  dependence:  a  multicenter 
trial  of  bupropion  for  cocaine  dependence  in  methadone-maintained  patients.  Journal  of  Drug 
and  Alcohol  Dependence  40:125-131,  1995. 

48.  Compton  PA.  Ling  W.  Wesson  DR.  Charuvastra  VC.  Wilkins  J.  Urine  toxicology  as  an  outcome 
measure  in  drug  abuse  clinical  trials:  must  every  sample  be  analyzed?  Journal  of  Addictive 
Diseases.  15(2):  85-92,  1996. 

49.  Shaner  A.  Roberts  LJ.  Eckman  TA.  Tucker  DE.  Tsuang  JW.  Wilkins  JN.  Mintz  J.  Monetary 
reinforcement  of  abstinence  from  cocaine  among  mentally  ill  patients  with  cocaine  dependence. 
Psychiatric  Services.  48(6):807-10,  1997. 

50.  Matthias  P.  Tashkin  DP.  Marques-Magallanes  JA.  Wilkins  JN.  Simmons  MS.  Effects  of  varying 
marijuana  potency  on  deposition  of  tar  and  delta9-THC  in  the  lung  during  smoking. 
Pharmacology,  Biochemistry  &  Behavior.  58(4):  1 145-50,  1997. 

5 1 .  Wilkins  JN.  Ashofteh  A.  Setoda  D.  Wheatley  WS.  Huigen  H.  Ling  W.  Ultrafiltration  using  the 
Amicon  MPS-1  for  assessing  methadone  plasma  protein  binding.  Therapeutic  Drug  Monitoring. 
19(l):83-7,  1997. 

52.  Wilkins  JN.  Pharmacotherapy  of  schizophrenia  patients  with  comorbid  substance  abuse. 
Schizophrenia  Bulletin.  23(2):2 15-28,  1997. 
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